diff --git a/API.md b/API.md index eff342b..5848dd0 100644 --- a/API.md +++ b/API.md @@ -105,7 +105,7 @@ service user-api { - 来源标签中的 `optional` / `omitempty`、标量 `default`、标量 `options=a|b`、数值 `range=[1:100]`（支持开区间和单边界）。 - `binding` 的 `required`、`min`、`max`、`len`、`email`。长度按 Unicode 字符或切片元素数；范围边界必须有限且位于 ±2^53 内。 -暂不支持匿名嵌入、别名、递归类型、外部 Go 类型、自定义 JSON 编解码、`[]byte`、泛型、`json:",string"`、任意 validator 标签、go-zero 的 JWT/签名/SSE 声明，以及客户端 SDK 生成。认证验证通过声明关联的命名中间件实现。未列出的语法会明确报错。 +暂不支持匿名嵌入、别名、递归类型、未声明的外部 Go 类型、自定义 JSON 编解码、`[]byte`、泛型、`json:",string"`、任意 validator 标签、go-zero 的 JWT/签名/SSE 声明，以及客户端 SDK 生成。认证验证通过声明关联的命名中间件实现。未列出的语法会明确报错。 业务错误、具名鉴权、多行文档及下述开发命令从 CLI `v0.7.0-preview.1` 提供。公共响应模板从 `v0.8.0-preview` 提供，CLI 与业务项目中的框架依赖需一起升级。 @@ -212,11 +212,21 @@ go test ./... 路由总览的 `(undeclared)` 仅表示没有声明 auth；旧式中间件仍可能做认证。 外发文档可用 export；项目内受生成器管理的 `docs/openapi.json` 请通过 generate 更新，以同步文件哈希。 +## 迁移扩展（v0.9.0-preview） + +- 动态 JSON 字段 `any` / `interface{}`，包含切片和 map；外围契约仍严格。 +- `scalar Name (...)` 显式引用项目类型，声明 JSON 输入/输出类型及文本绑定；支持已有 JSON/Text 转换方法。 +- `@body (mode: json/raw/multipart; maxBytes: ...)`：按路由限制请求体。 +- `@binding custom` 生成保留的绑定 hook；`@handlerMode manual` 生成保留的整个 Handler，raw/multipart 使用这个入口。 +- manual 可用 `@produces "text/plain"` 等声明无 DTO 成功输出。默认状态 200，初始手写 stub 返回 501。 + +完整语法、边界和示例见 [迁移扩展指南](MIGRATION.md)。新增生成代码要求 CLI 与框架依赖一起升级到 `v0.9.0-preview`；不兼容全部 goctl 语法。 + ## 参数语义 无 optional/default 的字段按“是否出现”判断必填，`false`、`0`、`""` 是已提供的值。额外声明 `binding:"required"` 才会拒绝标量零值。默认值只在字段缺失时使用，不替换显式的空值。 -业务需要区分“没传”和“传了零值”时使用 `*T` 加 optional。指针允许 JSON null；非指针拒绝 null；`binding:"required"` 拒绝指针 null。缺失与 null 都会得到 nil，本版没有三态 Nullable 类型。响应端的 `optional` 不等于 Go 编码器的 `omitempty`；需要省略零值时显式使用 `omitempty`。 +业务需要区分“没传”和“传了零值”时使用 `*T` 加 optional。指针允许 JSON null；普通非指针字段拒绝 null（动态 `any` 允许）；`binding:"required"` 拒绝指针 null。缺失与 null 都会得到 nil，本版没有三态 Nullable 类型。响应端的 `optional` 不等于 Go 编码器的 `omitempty`；需要省略零值时显式使用 `omitempty`。 JSON 必须是一个 object，Content-Type 为 application/json。默认最多 1 MiB，包含 chunked/未知长度和尾部空白；超限返回 413。重复 JSON key、未知字段、多个 JSON 值、标量 query/header 重复值、格式和约束错误返回 400；错误 Content-Type 返回 415。失败不会部分修改目标 DTO；错误详情只保留在内部 cause，公共响应不回显原始参数。 diff --git a/CHANGELOG.md b/CHANGELOG.md index 11e8cde..e2afabb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## v0.9.0-preview + +- 标准 HTTP Handler/Middleware 桥接：跟踪 Raw 状态与提交，保留取消、流式能力和中间件跨请求状态；已提交响应不追加 JSON。 +- `.api` 支持动态 JSON `any` / `interface{}`，保持重复键、大小、外围未知字段校验和数字精度。 +- 路由级请求体限额、自定义绑定和 manual Handler；上传、原始 body、下载可保留手写实现，重生成受保护。 +- 自定义 scalar 显式对接项目 JSON/Text 方法；支持可空和集合，分别生成请求/响应 OpenAPI；不放宽其他字段。 +- 新增 [迁移扩展指南](MIGRATION.md)、可运行示例和真实生成项目回归。CLI 与框架依赖一起升级。 + ## v0.8.0-preview - 公共响应 `codeType` 支持 `int32` / `int64` / `string`，校验默认码及映射范围，OpenAPI 标注整数位宽；JSON/YAML 和生成代码保留 int64 精度，旧 `integer` 兼容。 diff --git a/MIGRATION.md b/MIGRATION.md new file mode 100644 index 0000000..321d95b --- /dev/null +++ b/MIGRATION.md @@ -0,0 +1,182 @@ +# 已有 HTTP 项目逐步接入 + +从 `v0.9.0-preview` 开始，以下能力贯通运行时、`.api`、代码生成和 OpenAPI。CLI 和项目框架依赖一起升级；入口见 [README](README.md)。这些都是通用扩展点，框架不内置任何项目的 ID、支付回调、鉴权或数据库规则。 + +## 1. 保留标准 Handler 与中间件 + +```go +app := omni.New(omni.WithLogger(true)) +app.Use(httpx.WrapMiddleware(existingMiddleware)) // func(http.Handler) http.Handler +// go-zero 风格的 func(http.HandlerFunc) http.HandlerFunc： +app.Use(httpx.WrapFuncMiddleware(existingFuncMiddleware)) +app.POST("/legacy", httpx.WrapHandler(http.HandlerFunc(existingHandler))) +``` + +实际使用时先配置所有 `Use`，再注册路由。`WrapHandler` 也接受 `http.FileServer` 等标准 `http.Handler`。 + +- Handler 经过 App 的日志、requestId、恢复和应用中间件链，原来的 URL、状态码、Header、响应内容由它自己控制。 +- `c.Raw()` 的写入也会更新 `c.Status()` / `c.Responded()`；404 会记为 404。提交响应后返回错误或 panic，不会再追加错误 JSON。103 等临时响应不算最终提交。 +- 桥接器不缓存完整响应。下载继续流式复制；底层支持时保留 Flusher、Hijacker、Pusher、ReaderFrom、CloseNotifier。`http.ResponseController` 可沿 `Unwrap` 访问底层控制能力。能力仍取决于网络适配器和其他中间件。 +- 请求 context、取消信号和已有值继续传递，Omni 的路径参数可用 Go 1.22 的 `r.PathValue("id")` 读取。其他框架私有的参数读取器仍需由项目适配。标准中间件应从原 `r.Context()` 派生上下文；不能换成无关的 Background，也不能在 ServeHTTP 返回后调用 next。异步 timeout middleware 不在此桥接器的契约内。 +- 标准中间件在每条注册链构造一次，状态会跨请求保留。下游返回的错误在标准中间件退出前编码，因此压缩、审计等中间件能处理错误响应。它们自己的缓冲或改写行为仍由项目负责。 + +调用流使用 `App.Handle/GET/POST` 等入口即可，不要绕到原始 Router 注册。暂不使用 `.api` 的旧接口也能这样接入。 + +## 2. 动态 JSON：允许渐进补全 DTO + +```go +type LegacyReply { + Data any `json:"data"` +} +``` + +`interface{}` 是 `any` 的等价写法，支持 `[]any`、`map[string]any` 和对应的 `interface{}` 写法。带方法的 interface 不支持，会提示明确的类型错误。 + +动态字段允许 object、array、string、number、bool、null；OpenAPI 字段 schema 为 `{}`，不假装知道内部结构。它放开的是**该字段的内容**，外围 DTO 未声明的字段仍被拒绝，JSON 重复键、深度、总大小限制仍生效。动态数字解析为 `json.Number`，避免大整数先转 float64 丢精度。原始 JSON 的空白、转义和字节布局不会保留；验签请走原始 body 入口。 + +这不会自动补齐缺少的业务路由，也不代表兼容全部 goctl 语法。能确定结构的新接口优先使用命名 DTO。 + +## 3. 按路由限制 body 与保留手写入口 + +普通 JSON 路由只声明需要的大小，不改变其他路由默认的 1 MiB： + +```text +@body (mode: json + maxBytes: 2097152 +) +@handler createItem +post /items (CreateItemReq) returns (ItemResp) +``` + +`maxBytes` 单位为字节，允许 1 至 1 TiB；不写时为 1 MiB。已知超长或读取未知长度/chunked body 超限都按 413 处理。内置 App/RouterGroup 在全局中间件读取前安装 reader 限额；同时保持接口响应策略。网络层若有更小限额，以更小值为准。手写 Handler 吞掉读取错误或已经提交响应后，框架不能重新改写 HTTP 状态，务必检查读取错误。 + +### 只替换绑定函数 + +```text +@binding custom +@body (mode: json + maxBytes: 65536 +) +@handler createItem +post /items (CreateItemReq) returns (ItemResp) +``` + +首次生成 `internal/handler//createItemBind.go`： + +```go +func (h *Handler) bindCreateItem(c *httpx.Context, req *types.CreateItemReq) error { + // 调用项目自己的解码器，填充 req；失败返回 error。 +} +``` + +Handler 继续调用 Service 并使用统一响应；只把绑定过程交给这个函数。初始 hook 返回 501，不能忽略实现。自定义绑定需自己实现 Content-Type、未知字段、尾部数据等校验，并遵守 DTO 文档；框架负责路由总大小限制，不隐式套第二次 JSON 绑定。 + +### 上传、原文验签、下载 + +```text +@handlerMode manual +@body (mode: multipart + maxBytes: 10485760 +) +@handler upload +post /files + +@handlerMode manual +@body (mode: raw + maxBytes: 65536 +) +@produces "text/plain" +@handler callback +post /callback + +@handlerMode manual +@produces "application/octet-stream" +@handler download +get /download +``` + +首次生成 `Manual.go`，由项目实现整个 `func(h *Handler) Callback(c *httpx.Context) error`；依赖通过 `h.deps` 访问。生成器保留路由、中间件与文档管理，不会预读、JSON 解码或重新包装手写输出，也不生成这个路由的 Service 方法。需要调用 Service 时自行组合。 + +- manual 无返回类型默认成功状态 200；也可 `returns (UploadResp)` 描述 JSON DTO，手写实现按声明调用 `c.OK` 等。`@produces` 用于无 DTO 的手写输出，描述成功响应媒体类型，不会自动设置运行时 Header。 +- raw 的文档为二进制请求体；multipart 为开放对象，具体文件字段与逐部分校验由项目实现。raw/multipart 请求 DTO 仅可含 path/form/header 字段，不能同时声明 JSON body 字段。`form` 始终指 URL query。 +- 原文验签直接读取 `c.Request().Body`，在声明的小限额内 `io.ReadAll`；大文件用 `MultipartReader` 配合 `io.Copy` 流式处理，不能整个读入内存。 +- 使用 `ParseMultipartForm` 时，其内存参数只是内存阈值，不是总上传限额。App/桥接器在请求结束（含错误和 panic）后删除临时文件；业务仍须关闭 `FormFile` 得到的文件。`MultipartReader` 不自动建临时文件，业务自行创建的文件由业务清理。 +- 手写 Handler 的成功输出由它控制；返回的 error、鉴权拒绝仍走接口的错误策略。三方若要求错误也返回指定文本，使用 [自定义 Encoder](RESPONSE.md#手写路由配置文件与三方回调) 并维护对应错误文档；`@produces` 不会改变错误协议。 + +直接注册 Go 路由： + +```go +app.HandleWithBodyLimit("POST", "/upload", 10<<20, nil, uploadHandler) +// nil encoder 沿用 App 策略，也可传项目自己的 response.Encoder。 +``` + +通用注册器使用 `httpx.HandleWithBodyLimit(...)`；实现 `httpx.BodyLimitRegistrar` 可像 App 一样在全局中间件前限流量，未实现时只能在路由中间件入口限制。单独 `httpx.LimitBody` 同样受中间件顺序影响。 + +### 重新生成规则 + +`handlersGen.go`、`routesGen.go`、DTO、OpenAPI 继续由生成器管理；`*Bind.go`、`*Manual.go` 只首次创建，后续不会覆盖或删除。切换 manual 回自动模式时，先迁移实现并自行移除对应 Manual 文件，否则生成器明确拒绝，避免重复方法。修改 DTO 导致手写函数签名不匹配时，需要一起调整项目代码；`go test ./...` 是必要检查。 + +## 4. 自定义标量：项目定义行为，框架连接调用 + +例如输入 ID 兼容数字/字符串、输出统一为字符串： + +```text +scalar BusinessID ( + goImport: "example.com/billing/internal/ids" + goType: ID + jsonInput: string | int64 + jsonOutput: string + text: true +) +type ItemReq { + ID BusinessID `path:"id"` +} +type ItemResp { + ID BusinessID `json:"id"` +} +``` + +`goImport`/`goType` 指向项目已有的具体 Go 类型，不是框架内置 ID。生成器创建类型别名和 `types.ContractScalars()`，然后把注册表显式传给每个 Binder。运行时使用标准接口： + +| 项目类型的方法 | 用途 | +| --- | --- | +| 值接收者 `MarshalJSON() ([]byte, error)` | JSON 输出 | +| 指针接收者 `UnmarshalJSON([]byte) error` | JSON 输入 | +| 指针接收者 `UnmarshalText([]byte) error` | `text: true` 时的 path/query/header | + +完整参考 [示例 ID 类型](examples/compatibility/ids/id.go)。CLI 校验声明并生成代码，不执行项目 Go 代码；启动时校验方法是否齐全，缺失会明确失败。 + +输入可选 `string`、`bool`、`int32`、`int64`、`float64`，用 `|` 声明多个允许的 JSON 类型；输出只声明一个。框架先校验 JSON 类型和数字范围，再调用项目转换方法。字符串是否是合法 ID、是否允许空串或负数由项目方法决定，不能把数字溢出静默截断。path/query/header 实际传输文本，OpenAPI 对这些位置描述 string；query 切片通过重复键传递。 + +普通字段不受注册影响，普通 `int64` 仍拒绝 JSON 字符串。没有全局注册表，也不会自动给所有 ID 放宽规则。标量可嵌套、放入切片/map，`*BusinessID` 允许 null；缺失默认必填，`optional`/`omitempty` 可选。标量的额外约束放在转换器中，目前不叠加 `binding`、default/options/range 等标签。 + +请求 OpenAPI 按 `string | int64` 生成 `anyOf`，响应按 `string` 生成，避免把输入兼容误描述为输出也有两种类型。输出方法必须遵守声明，框架不会每次重新验证响应；项目需为自定义类型测试边界行为。 + +不使用 `.api` 也可显式注册： + +```go +binder := contract.MustNew[CreateReq]( + contract.ScalarOf[ids.ID]("string|int64", "string", true), +) +// binder.Bind(c.Request(), httpx.RouteParams(c.Context()), &req) +``` + +## 可运行示例与验收 + +```sh +go run ./examples/compatibility +curl -i http://127.0.0.1:8080/legacy +curl -i -X POST http://127.0.0.1:8080/echo \ + -H 'Content-Type: application/json' -d '{"id":9223372036854775807,"data":{"hello":true}}' +curl -i -F file=@README.md http://127.0.0.1:8080/upload +curl -OJ http://127.0.0.1:8080/download +``` + +完整契约：[compatibility.api](examples/contracts/compatibility.api)。生成项目起初的手写 hook 仍为 501；真实实现与原文 HMAC 验签回归见 [消费者 fixture](cmd/go-omni/apigen/testdata/compatibility)。 + +```sh +GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race ./cmd/go-omni/apigen -run Compatibility +go test -race ./httpx ./contract +``` + +这些是框架验收，不代表任何业务项目已经迁移完毕；业务的路由覆盖、响应兼容、权限、资金事务和数据库行为需要业务自己的回归。 diff --git a/README.md b/README.md index 8f2ce85..9ba8c78 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ GoOmni 是面向多个业务项目复用的 Go 后端框架：用 `.api` 定义接口、请求和返回类型，生成项目目录、HTTP 接入代码及 OpenAPI，业务代码负责 Service 和 Repository。也可以在已有项目中直接使用 `omni.New()`，或单独引入日志、存储等模块。 -**从这份 README 开始即可接入。** 当前版本为 **`v0.8.0-preview`**，Go **1.22+**；CLI 和项目依赖使用同一固定版本。框架不依赖 Gin，默认使用标准库网络层，提供可切换的 native HTTP/1.1 和实验性 HTTP/2 适配器。 +**从这份 README 开始即可接入。** 当前版本为 **`v0.9.0-preview`**，Go **1.22+**；CLI 和项目依赖使用同一固定版本。框架不依赖 Gin，默认使用标准库网络层，提供可切换的 native HTTP/1.1 和实验性 HTTP/2 适配器。 响应字段、业务码、登录方式、权限规则和数据库表结构由接入项目决定。文中的 `code/msg/data`、`200/9999` 都是示例配置，可以替换。 @@ -12,6 +12,7 @@ GoOmni 是面向多个业务项目复用的 Go 后端框架：用 `.api` 定义 | --- | --- | | 新建服务，先定义接口再开发 | [新项目：生成并运行](#新项目生成并运行) | | 已有服务，逐步接入 HTTP 或基础模块 | [已有项目：按模块接入](#已有项目按模块接入) | +| 已有标准 Handler、动态 JSON、上传或自定义 ID 类型 | [迁移扩展指南](MIGRATION.md) | | 想统一返回体、业务错误或三方回调 | [配置公共响应与独立协议](#配置公共响应与独立协议) | | 要接日志、数据库、鉴权和存储 | [业务开发与依赖装配](#业务开发与依赖装配) | | 要切换网络层、配置 TLS 和退出清理 | [配置与运行](#配置与运行) | @@ -25,7 +26,7 @@ module 地址是 `git.yuntuo.hk/k2safe/go-omni`。先确保本机 Go 和 Git 可 ```sh export GOPRIVATE=git.yuntuo.hk -go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.8.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.0-preview export PATH="$(go env GOPATH)/bin:$PATH" go-omni --help ``` @@ -37,7 +38,7 @@ go-omni --help 在准备存放业务项目的目录执行，`./hello` 必须是新目录或空目录；把 `example.com/hello` 换成自己的 Go module 名称。 ```sh -go-omni api new --framework-version v0.8.0-preview \ +go-omni api new --framework-version v0.9.0-preview \ --module example.com/hello ./hello cd hello go mod tidy @@ -125,7 +126,7 @@ service catalog-api { ```sh go-omni api validate --api catalog.api -go-omni new --api catalog.api --framework-version v0.8.0-preview \ +go-omni new --api catalog.api --framework-version v0.9.0-preview \ --module example.com/catalog ./catalog cd catalog go mod tidy @@ -166,21 +167,21 @@ curl -i http://127.0.0.1:8080/api/v1/items/missing 需要 MySQL、文件存储或文件日志时，**首次生成**可以使用： ```sh -go-omni new --api catalog.api --framework-version v0.8.0-preview \ +go-omni new --api catalog.api --framework-version v0.9.0-preview \ --module example.com/catalogFull --database mysql --storage local \ --log-output file --log-format json ./catalogFull ``` 该命令在 `catalog.api` 所在目录执行，目标仍需是新目录。数据库迁移、表结构和 Repository 由项目维护，API 模板不自动建表；MySQL DSN 通过 `GO_OMNI_DB_DSN` 提供。存储还可选 `s3`、`r2`、`oss`，按生成项目的配置说明提供桶、端点和凭据。完整装配例子见 [userService](examples/userService/README.md)。 -不采用 `.api` 时，也可以用 `go-omni new --framework-version v0.8.0-preview --module example.com/billing ./billing` 创建手写接口的 CRUD 示例项目；两种生成方式选择一种即可。 +不采用 `.api` 时，也可以用 `go-omni new --framework-version v0.9.0-preview --module example.com/billing ./billing` 创建手写接口的 CRUD 示例项目；两种生成方式选择一种即可。 ## 已有项目：按模块接入 在业务项目的 **go.mod 所在目录**执行： ```sh -go get git.yuntuo.hk/k2safe/go-omni@v0.8.0-preview +go get git.yuntuo.hk/k2safe/go-omni@v0.9.0-preview ``` 已有项目可直接复用现有目录、Service、Repository 和数据库连接，先选择一个接口接入。也可以只用 `logx`、`storage` 等包，暂时保留原来的 HTTP 框架。 @@ -243,11 +244,13 @@ func run() error { 运行 `go run .`，访问 `/api/v1/items/1001`、`/api/v1/items/missing` 和 `/ping`，可分别验证成功、业务错误及纯文本输出。`App` 实现标准 `http.Handler`，也可挂进现有 `http.Server` 或父路由；这时监听、TLS、信号和资源关闭由现有入口负责，结束时调用 `app.Close()`。 +现有 `http.HandlerFunc` 可直接通过 `app.POST("/legacy", httpx.WrapHandler(http.HandlerFunc(oldHandler)))` 接入；状态码与访问日志一致，下载和流式输出保持原协议。标准中间件、动态 JSON、上传/原文 body、手写绑定和自定义类型的完整用法见 [迁移扩展指南](MIGRATION.md)。 + 接入现有业务时，逐项确认： - **协议**：保留既有 URL、HTTP 状态、字段、分页和空值语义。已有业务错误类型需要在项目内转换为 Omni 错误，或使用 `WithResponseEncoder` 适配原错误；设置字段模板不会自动识别其他库的错误。 - **鉴权**：把现有登录态和权限校验接入中间件，继续使用原来的授权范围和业务规则。原来依赖 `gin.Context` 等类型的 HTTP 代码需要适配；标准 `context.Context` 的业务服务可以继续复用。 -- **数据类型**：ID、金额、日期等按项目协议转换。`.api` 当前不支持外部 Go 类型、自定义 JSON 编解码及 `json:",string"`；需要 JSON 字符串时在契约中声明 `string`，在业务层显式转换。 +- **数据类型**：ID、金额、日期等按项目协议转换。用 `.api` 的 `scalar` 显式关联项目自己的类型与 JSON/Text 转换方法，分别声明输入和输出协议；普通字段保持严格绑定，见 [自定义标量](MIGRATION.md#4-自定义标量项目定义行为框架连接调用)。 - **生成器**：它不会自动接管任意已有 Go 项目。希望引入 `.api` 时，先在独立新目录生成对照，再规划路由与生成目录的接入，不要用脚手架覆盖现有业务目录。 先通过一个接口的成功、参数错误、未登录、无权限及业务错误回归，再按模块迁移。框架不要求所有项目使用同一套业务目录或身份模型。 @@ -273,7 +276,7 @@ func run() error { **生成接口的格式以 `.api` 为准**，修改后重新生成，运行时与 OpenAPI 一起更新；应用 YAML 不会覆盖它。404/405 没有匹配到某个 `.api` 接口，使用应用默认配置，需要统一时也配置应用默认值。 -文本/XML 回调用 Go 编码器或直接输出，并维护对应文档；`.api` 的 `raw` 仅表示原始 JSON。前面示例的 `c.Text` 控制这次正常返回；需要连鉴权失败、业务错误和 panic 都按回调协议响应时使用 `HandleWithResponse`。完整配置和回调示例见 [响应参考](RESPONSE.md)。 +文本/XML 回调用 Go 编码器或直接输出，`.api` 可用 `@handlerMode manual` 与 `@produces` 声明成功媒体类型；特殊错误协议仍需项目维护。`@response raw` 仅表示原始 JSON。前面示例的 `c.Text` 控制这次正常返回；需要连鉴权失败、业务错误和 panic 都按回调协议响应时使用 `HandleWithResponse`。完整配置和回调示例见 [响应参考](RESPONSE.md)。 ## 业务开发与依赖装配 @@ -361,7 +364,7 @@ go test ./... `Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理，手动修改会报告冲突。Service、Repository、中间件实现、main、配置和依赖装配属于项目，重复生成保留手写内容；契约改变后仍需自行调整业务签名并通过编译。将 `.api`、生成文件和 `.goOmni/apiManifest.json` 一起提交，CI 可运行 `generate --check` 检查是否同步。 -升级时先安装目标版本的 CLI，再在业务 module 中 `go get` **同一版本**；API 项目随后执行上述生成与测试步骤，手写项目只需依赖升级及业务回归。`v0.8.0-preview` 的变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。 +升级时先安装目标版本的 CLI，再在业务 module 中 `go get` **同一版本**；API 项目随后执行上述生成与测试步骤，手写项目只需依赖升级及业务回归。`v0.9.0-preview` 的变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。 ## 按需深入 diff --git a/RELEASE.md b/RELEASE.md index 1dc11d2..2baa818 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,46 +1,34 @@ -# v0.8.0-preview +# v0.9.0-preview -这一版让业务自行定义公共返回体：在 `.api` 中配置字段、结构、成功/失败提示和错误码类型，按分组复用或由单接口覆盖，运行时与 OpenAPI 共用定义。需要特殊协议的回调可独立控制状态、Header 和内容。 - -本版继续作为预览版供固定版本接入；HTTP/2 仍是独立实验适配器。 +本版补齐已有 HTTP 项目的迁移接入能力：标准 Handler/Middleware 桥接、动态 JSON、路由级请求体限额与手写绑定、自定义标量转换。业务的响应格式、身份、资金事务和数据库实现仍由项目决定。 ## 安装与升级 -首次接入请从 [README 接入指南](README.md) 开始，按新项目或已有项目选择步骤。下面是已接入项目的版本升级命令： - ```sh -GOPRIVATE=git.yuntuo.hk go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.8.0-preview - -# 在业务项目 go.mod 所在目录执行： -GOPRIVATE=git.yuntuo.hk go get git.yuntuo.hk/k2safe/go-omni@v0.8.0-preview -# 仅在已经由 .api 生成、具有 .goOmni/apiManifest.json 的项目中执行： -go-omni api generate -go test ./... +export GOPRIVATE=git.yuntuo.hk +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.0-preview +# 在业务 module 中： +go get git.yuntuo.hk/k2safe/go-omni@v0.9.0-preview ``` -手写接口项目跳过 `api generate`，仍需执行业务测试。生成器不会自动接管其他框架创建的已有项目。 +将私有域名追加到已有 GOPRIVATE；凭据通过本机 Git 凭据管理器配置。CLI 与项目框架依赖一起升级。进入已生成的项目，再运行 `go-omni api generate --dry-run`、`go-omni api generate`、`go test ./...`。 -新项目可以直接 `go-omni api new --module example.com/demo ./demo`，或用 `go-omni new --api service.api --module example.com/demo ./demo`。在框架源码目录之外运行，远程安装的 CLI 会固定生成项目的框架版本，无需本地 replace；也可显式传 `--framework-version v0.8.0-preview` 固定远程依赖。需要仓库权限时使用自己的 Git 凭据管理器，不把凭据写入 module path 或 go.mod。 +## 本次变化 -## 本版功能 +- `httpx.WrapHandler`、`WrapMiddleware`、`WrapFuncMiddleware` 将现有标准代码接入 App 中间件链。Raw 写入会记录真实状态和提交；已经输出的响应不再追加错误 JSON。下载/流式输出不整体缓存，保留底层支持的 HTTP 可选能力和取消信号。 +- `.api` 的 `any` / `interface{}` 支持动态 JSON。数字保留为 json.Number，外围 DTO 严格校验不变。 +- `@body` 按路由声明限额；内置 App 在全局中间件读取前限制 body。`@binding custom`、`@handlerMode manual` 生成保留的项目实现文件；上传、原文验签、下载可自行处理。`@produces` 描述手写成功输出的媒体类型。 +- `scalar` 显式引用项目 Go 类型，调用标准 JSON/Text 方法；输入和输出各自生成 OpenAPI。普通基础类型不会自动放宽。 -- `response Name (...)` 定义公共 JSON 体，支持嵌套结构、独立 errorBody、常量及类型保持的占位符；通过 import 共享配置。 -- `@server (response: Name)` 为整个 service 块选择模板，`@response Name` 单接口覆盖；`@response raw` 成功返回业务 JSON，错误保留组级格式。 -- `codeType: int32 / int64 / string` 使用与业务字段一致的名称；默认码和映射都校验，int64 配置及生成代码保留整数精度。旧 `integer` 写法仍兼容。 -- `successMessage` / `errorMessage` 配置默认提示，`codes` / `messages` 按逻辑错误码覆盖。业务只返回数据或错误，公共体由框架组装。 -- `omni.WithResponse` 设置手写路由的应用默认格式，`WithResponseEncoder` 与 `HandleWithResponse` 支持特殊协议；Text / XML / Bytes / JSON / NoContent 可直接输出。 -- 生成路由在全局中间件之前选定策略，提前鉴权拒绝、限流、panic 和 Context 克隆沿用正确格式。OpenAPI 同步模板结构与业务错误示例。 +详见 [迁移扩展指南](MIGRATION.md)、[完整契约](examples/contracts/compatibility.api)、[运行示例](examples/compatibility/main.go) 和 [历史更新](CHANGELOG.md)。 -详见 [公共响应体](RESPONSE.md)、[完整定义示例](examples/contracts/responses.api) 和 [更新记录](CHANGELOG.md)。 +## 兼容与边界 -## 升级行为 +原有 `.api`、公共响应配置和普通 Handler 可以继续使用。新生成代码最低运行时版本为本版。`Raw()` 现在返回跟踪包装器；直接断言某个具体 writer 类型的代码需改用标准接口/ResponseController。 -1. CLI 与框架依赖一起升级到本版。新生成路由使用新的响应策略入口，旧运行时不能编译。旧 `.api` 可继续使用；生成器不会覆盖已有 Service、Repository、middleware、配置和 main。 -2. 生成接口以 `.api` 为响应契约来源，即使没有声明模板也固定使用默认格式；应用 YAML 的 response 用于手写路由和应用兜底。需要变更生成接口的格式，请修改 `.api` 后重新生成。 -3. `response.Response` DTO 保留为兼容类型，运行时不再依赖它固定结构。未配置的正常业务响应继续使用默认 code/message/data/requestId。 -4. 内置 App 的 404/405 现在经过应用响应策略和中间件，默认输出 JSON；原先依赖纯文本兜底的客户端需调整。直接使用 nethttp.Router 的兜底行为不变。 -5. `int32` / `int64` 按有符号整数范围校验并输出 JSON 数值；消费端需支持相应精度。需要文本码时使用 string。旧 integer 继续保留原有 ±(2^53−1) 范围。 -6. `.api` 的模板和 raw 当前描述 JSON。文本/XML回调用 Go 编码器并维护对应文档；TLS/HTTP解析错误、连接级过载等发生在进入应用前的错误仍按网络协议处理。 +标准中间件须保留 incoming request context 并同步调用 next；第三方框架私有的参数读取器仍由项目适配。手写 Handler/绑定 hook 初始返回 501；实现和文档一致性由项目负责，未实现不能上线。切换 manual 回自动模式时须显式迁移并移除对应手写文件，生成器不会删除它。 + +新增能力没有引入 Gin 或 go-zero 依赖。不是完整 goctl 语法兼容层，不代表任何业务项目已完成迁移。HTTP/2 仍为实验功能，本轮没有重新做 Linux 长测、代理验收或云存储账号实网验收。 ## 验证入口 @@ -48,13 +36,8 @@ go test ./... GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=1 -count=1 -timeout=300s ./... go vet ./... go build ./... -python3 scripts/verifyAPIWorkflow.py --report performance/responsePoliciesWorkflowLocal.json -python3 scripts/verifyResponseRelease.py --version v0.8.0-preview \ - --report performance/responsePoliciesRemote.json -python3 scripts/verifyAPIWorkflow.py --version v0.8.0-preview \ - --report performance/responsePoliciesWorkflowRemote.json +python3 scripts/verifyCompatibilityRelease.py --version v0.9.0-preview \ + --report performance/compatibilityRemote.json ``` -远程验收分别从全新 module cache 安装 CLI，验证生成项目固定依赖且没有本地 replace，以及完整用户示例的真实 MySQL、nethttp/native、事务、日志和生命周期。数据库容器只使用本机临时资源（1 CPU / 768 MiB），结束清理，不连接业务服务器。 - -封版验证完成：40 个包全量 race 通过；全新缓存安装的远程 CLI/框架、公共模板消费者及 nethttp/native 的 34 项业务检查通过，无本地 replace。验收工具和最终记录维护在 main，发布标签保持固定。详见 [响应模板验收](performance/responsePoliciesAcceptance.md)。OSS/R2 云账号实网验收与长时间压力测试不属于本次发布范围。 +测试覆盖旧 Handler 404 与访问日志、提交后错误/panic、标准中间件状态/取消/提前写响应、nethttp/native 实际流式下载与请求体超限、multipart 临时文件清理、自定义标量精度/越界/nullable/集合，以及真实生成项目的验签与重生成保护。远程脚本从全新 module cache 安装 CLI，并检查生成项目的固定版本、校验和以及没有本地 replace。 diff --git a/RESPONSE.md b/RESPONSE.md index 89ff8b6..f5e4835 100644 --- a/RESPONSE.md +++ b/RESPONSE.md @@ -125,7 +125,7 @@ app.HandleWithResponse("POST", "/hooks/payment", ack, paymentHandler) 编码器也可以设置 HTTP 状态、Header、XML 或任意字节。`omni.WithResponseEncoder` 设置应用默认编码器；`httpx.WithResponse` 是普通路由中间件，按中间件顺序生效。需要覆盖全局中间件提前拒绝时，应使用 `HandleWithResponse`。自定义 Registrar 可以实现 `httpx.ResponseRegistrar` 提供相同的早期挂载能力；未实现时辅助函数退化为普通路由中间件。 -`.api` 当前声明的是 JSON 协议，`@response raw` 也是原始 JSON，不代表任意文本/XML协议。自定义文本/XML回调使用上述 Go 入口，并由应用维护相应文档；生成器不会猜测自定义 Go 编码器的输出。如果在命名中间件里替换编码器，业务也必须同步维护文档。 +`@response raw` 表示原始 JSON，不代表任意文本/XML协议。从 v0.9.0-preview 开始，`.api` 可用 `@handlerMode manual` 配合 `@produces` 描述文本/文件等成功输出，详见 [迁移扩展](MIGRATION.md)。自定义回调错误协议仍使用上述 Go 入口，并由应用维护相应文档；生成器不会猜测自定义 Go 编码器的输出。如果在命名中间件里替换编码器，业务也必须同步维护文档。 框架内置路由的 404/405 使用应用默认格式；它们不属于某个 `.api` 分组。TLS 握手、畸形 HTTP、连接级过载等发生在进入应用前，仍按网络层协议处理。自定义编码器失败、panic 或返回非法状态/204 body 时，最终返回空 500，避免递归编码或泄露另一种响应体。 diff --git a/app/body.go b/app/body.go new file mode 100644 index 0000000..cae0eb8 --- /dev/null +++ b/app/body.go @@ -0,0 +1,32 @@ +package app + +import ( + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/response" + "net/http" +) + +func (a *App) bodyHandler(maxBytes int64, encoder response.Encoder, handler httpx.Handler, middleware ...httpx.Middleware) http.Handler { + checked := httpx.LimitBody(maxBytes)(handler) + if encoder == nil { + encoder = a.responseEncoder + } + next := a.adaptResponse(encoder, checked, middleware...) + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Body != nil { + r = r.WithContext(r.Context()) + r.Body = http.MaxBytesReader(w, r.Body, maxBytes) + } + next.ServeHTTP(w, r) + }) +} + +// HandleWithBodyLimit caps the reader before application middleware and preserves +// the response policy, including auth failures and parameter errors. +func (a *App) HandleWithBodyLimit(method, path string, maxBytes int64, encoder response.Encoder, handler httpx.Handler, middleware ...httpx.Middleware) { + a.router.Handle(method, path, a.bodyHandler(maxBytes, encoder, handler, middleware...)) +} +func (g *RouterGroup) HandleWithBodyLimit(method, path string, maxBytes int64, encoder response.Encoder, handler httpx.Handler, middleware ...httpx.Middleware) { + combined := append(append([]httpx.Middleware{}, g.middlewares...), middleware...) + g.group.Handle(method, path, g.app.bodyHandler(maxBytes, encoder, handler, combined...)) +} diff --git a/app/router.go b/app/router.go index 2d0b1a0..62cdcb6 100644 --- a/app/router.go +++ b/app/router.go @@ -31,6 +31,7 @@ func (a *App) adaptResponse(encoder response.Encoder, handler httpx.Handler, mid } requestContext := httpx.NewContext(request, writer, httpx.RouteParams(request.Context())) requestContext.SetResponseEncoder(encoder) + defer func() { httpx.CleanupMultipart(requestContext.Request()) }() if err := wrapped(requestContext); err != nil { a.writeError(requestContext, err) } diff --git a/cmd/go-omni/README.md b/cmd/go-omni/README.md index 412c55f..62834ae 100644 --- a/cmd/go-omni/README.md +++ b/cmd/go-omni/README.md @@ -75,8 +75,8 @@ Handler / Service / Repository / Model 与模块注册入口。模块名为 lowe 从远程安装，生成一个带 Handler / Service / Repository / Model 分层的服务骨架： ```sh -GOPRIVATE=git.yuntuo.hk go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.8.0-preview -go-omni new --framework-version v0.8.0-preview --module example.com/billing ./billing +GOPRIVATE=git.yuntuo.hk go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.0-preview +go-omni new --framework-version v0.9.0-preview --module example.com/billing ./billing cd billing go mod tidy go run ./cmd/server diff --git a/cmd/go-omni/apigen/compatibility_test.go b/cmd/go-omni/apigen/compatibility_test.go new file mode 100644 index 0000000..7cbd9f8 --- /dev/null +++ b/cmd/go-omni/apigen/compatibility_test.go @@ -0,0 +1,161 @@ +package apigen + +import ( + "context" + "encoding/json" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestCompatibilityContract(t *testing.T) { + s, err := Parse(filepath.Join(frameworkRoot(t), "examples/contracts/compatibility.api")) + if err != nil { + t.Fatal(err) + } + c, err := Compile(s) + if err != nil { + t.Fatal(err) + } + if len(s.Scalars) != 1 || len(s.Routes) != 6 { + t.Fatal("incomplete contract") + } + if !strings.Contains(string(c.Document), `"anyOf"`) || !strings.Contains(string(c.Document), `"multipart/form-data"`) { + t.Fatal("missing schemas") + } + for _, change := range [][2]string{{"jsonInput: string | int64", "jsonInput: mystery"}, {"text: true", "text: false"}, {"mode: multipart", "mode: invalid"}, {"maxBytes: 128", "maxBytes: 0"}, {"goType: ID", "goType: id"}, {"Data any", "Data interface{ Method() }"}} { + source := readFile(t, filepath.Join(frameworkRoot(t), "examples/contracts/compatibility.api")) + spec, err := parseText(t, strings.Replace(source, change[0], change[1], 1)) + if err == nil { + _, err = Compile(spec) + } + if err == nil { + t.Fatalf("accepted %s", change[1]) + } + } +} +func TestCompatibilityConsumer(t *testing.T) { + if os.Getenv("GO_OMNI_SCAFFOLD_INTEGRATION") == "" { + t.Skip("consumer integration") + } + root := frameworkRoot(t) + dir := filepath.Join(t.TempDir(), "consumer") + o := Options{API: filepath.Join(root, "examples/contracts/compatibility.api"), Dir: dir, Module: "example.com/compatibility", FrameworkDir: root} + releaseCLI := os.Getenv("GO_OMNI_RELEASE_CLI") + generate := func(check bool) { + t.Helper() + if releaseCLI == "" { + options := o + options.Check = check + if _, err := Generate(options); err != nil { + t.Fatal(err) + } + return + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + args := []string{"api", "generate", "--api", o.API, "--dir", dir, "--module", o.Module} + if check { + args = append(args, "--check") + } + cmd := exec.CommandContext(ctx, releaseCLI, args...) + cmd.Dir = filepath.Dir(dir) + if output, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("released CLI: %v\n%s", err, output) + } + } + generate(false) + files := map[string]string{ + "internal/service/demo/echo.go": "echo.go", + "internal/handler/demo/customBind.go": "hooks.go", + "internal/handler/files/downloadManual.go": "download.go", + "consumer_test.go": "consumer_test.go", + } + // The fixture implements related methods together; remove only its generated + // one-time stubs, then fill every stub so regeneration can preserve all files. + for _, name := range []string{"internal/service/demo/item.go", "internal/service/demo/custom.go"} { + writeFile(t, filepath.Join(dir, name), "package demo\n") + } + for _, name := range []string{"internal/handler/demo/uploadManual.go", "internal/handler/demo/callbackManual.go"} { + writeFile(t, filepath.Join(dir, name), "package demo\n") + } + for target, source := range files { + writeFile(t, filepath.Join(dir, target), readFile(t, filepath.Join("testdata/compatibility", source))) + } + before := map[string]string{} + for target := range files { + before[target] = readFile(t, filepath.Join(dir, target)) + } + generate(false) + for target, content := range before { + if readFile(t, filepath.Join(dir, target)) != content { + t.Fatalf("overwrote %s", target) + } + } + generate(true) + ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, "go", "test", "-mod=mod", "-race", "-count=1", "./...") + cmd.Dir = dir + cmd.Env = append(os.Environ(), "GOWORK=off") + if output, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("consumer: %v\n%s", err, output) + } + if releaseCLI != "" { + cmd = exec.CommandContext(ctx, "go", "list", "-m", "-json", frameworkModule) + cmd.Dir = dir + cmd.Env = append(os.Environ(), "GOWORK=off") + output, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("module: %v\n%s", err, output) + } + var module struct { + Version, Sum string + Replace any + } + if err = json.Unmarshal(output, &module); err != nil { + t.Fatal(err) + } + if module.Version != os.Getenv("GO_OMNI_RELEASE_VERSION") || module.Replace != nil || module.Sum == "" { + t.Fatalf("not a fixed remote dependency: %+v", module) + } + t.Logf("remote framework %s checksum %s; no local replace", module.Version, module.Sum) + } + +} + +func TestNestedDynamicAliases(t *testing.T) { + source := readFile(t, filepath.Join(frameworkRoot(t), "examples/contracts/compatibility.api")) + for _, typ := range []string{"[]interface{}", "map[string]interface{}", "map[string][]interface{}"} { + s, err := parseText(t, strings.Replace(source, "Data any", "Data "+typ, 1)) + if err == nil { + _, err = Compile(s) + } + if err != nil { + t.Fatalf("%s: %v", typ, err) + } + } +} +func TestSwitchFromManualRequiresExplicitMigration(t *testing.T) { + root := frameworkRoot(t) + o := Options{API: filepath.Join(root, "examples/contracts/compatibility.api"), Dir: filepath.Join(t.TempDir(), "consumer"), Module: "example.com/compatibility", FrameworkDir: root} + if _, err := Generate(o); err != nil { + t.Fatal(err) + } + // The hand-written source may contain business code. Never delete it to make + // the newly automatic handler compile; fail before touching generated files. + o.API = filepath.Join(o.Dir, "api/compatibility.api") + source := readFile(t, o.API) + source = strings.Replace(source, "@handlerMode manual\n @produces \"application/octet-stream\"", "", 1) + writeFile(t, o.API, source) + before := snapshot(t, o.Dir) + if _, err := Generate(o); err == nil || !strings.Contains(err.Error(), "migrate and remove") { + t.Fatalf("unsafe mode switch: %v", err) + } + if !sameFiles(before, snapshot(t, o.Dir)) { + t.Fatal("failed mode switch mutated project") + } +} diff --git a/cmd/go-omni/apigen/compile.go b/cmd/go-omni/apigen/compile.go index 00be95d..3101985 100644 --- a/cmd/go-omni/apigen/compile.go +++ b/cmd/go-omni/apigen/compile.go @@ -42,6 +42,9 @@ func Compile(spec *Spec) (*Compiled, error) { } definitions := map[string]Type{} for _, t := range spec.Types { + if t.Name == "ContractScalars" && len(spec.Scalars) > 0 { + return nil, errors.New("ContractScalars is reserved when declaring scalars") + } if !exportedName.MatchString(t.Name) { return nil, fmt.Errorf("%s: type names must be UpperCamelCase", t.Position) } @@ -50,17 +53,24 @@ func Compile(spec *Spec) (*Compiled, error) { } definitions[t.Name] = t } + scalarTypes, scalarDefs, err := compileScalars(spec, definitions) + if err != nil { + return nil, err + } resolved := map[string]reflect.Type{} resolving := map[string]bool{} var resolve func(string, int) (reflect.Type, error) var expression func(ast.Expr, int) (reflect.Type, error) - primitives := map[string]reflect.Type{"string": reflect.TypeOf(""), "bool": reflect.TypeOf(false), "int": reflect.TypeOf(int(0)), "int8": reflect.TypeOf(int8(0)), "int16": reflect.TypeOf(int16(0)), "int32": reflect.TypeOf(int32(0)), "int64": reflect.TypeOf(int64(0)), "uint": reflect.TypeOf(uint(0)), "uint8": reflect.TypeOf(uint8(0)), "uint16": reflect.TypeOf(uint16(0)), "uint32": reflect.TypeOf(uint32(0)), "uint64": reflect.TypeOf(uint64(0)), "float32": reflect.TypeOf(float32(0)), "float64": reflect.TypeOf(float64(0))} + primitives := map[string]reflect.Type{"any": reflect.TypeOf((*any)(nil)).Elem(), "string": reflect.TypeOf(""), "bool": reflect.TypeOf(false), "int": reflect.TypeOf(int(0)), "int8": reflect.TypeOf(int8(0)), "int16": reflect.TypeOf(int16(0)), "int32": reflect.TypeOf(int32(0)), "int64": reflect.TypeOf(int64(0)), "uint": reflect.TypeOf(uint(0)), "uint8": reflect.TypeOf(uint8(0)), "uint16": reflect.TypeOf(uint16(0)), "uint32": reflect.TypeOf(uint32(0)), "uint64": reflect.TypeOf(uint64(0)), "float32": reflect.TypeOf(float32(0)), "float64": reflect.TypeOf(float64(0))} expression = func(e ast.Expr, depth int) (reflect.Type, error) { if depth > 32 { return nil, errors.New("type nesting exceeds 32") } switch e := e.(type) { case *ast.Ident: + if t, ok := scalarTypes[e.Name]; ok { + return t, nil + } if t, ok := primitives[e.Name]; ok { return t, nil } @@ -140,7 +150,7 @@ func Compile(spec *Spec) (*Compiled, error) { if err != nil { return nil, err } - b, err := contract.ForType(rt) + b, err := contract.Describe(rt, scalarDefs...) if err != nil { return nil, fmt.Errorf("%s: %w", t.Position, err) } @@ -217,12 +227,15 @@ func Compile(spec *Spec) (*Compiled, error) { if r.Status != 200 && r.Status != 201 && r.Status != 202 && r.Status != 204 { return nil, errors.New("@status supports 200, 201, 202 or 204") } - if (r.Status == 204) != (r.Response == "") { + if !r.Manual && (r.Status == 204) != (r.Response == "") { return nil, errors.New("204 requires no response; other success statuses require a response type") } if r.Method == "head" && r.Response != "" { return nil, errors.New("HEAD must not declare a response body") } + if err := validateRouteExtensions(r); err != nil { + return nil, fmt.Errorf("%s: %w", r.Position, err) + } op := openapi.Operation{Method: r.Method, Path: r.Path, OperationID: operationID(r), Summary: r.Summary, Responses: map[int]openapi.Response{}} if r.Request != "" { b, ok := c.Bindings[r.Request] @@ -231,6 +244,9 @@ func Compile(spec *Spec) (*Compiled, error) { } op.Parameters = b.Parameters() op.Request = b.BodySchema() + if op.Request != nil && (r.BodyMode == "raw" || r.BodyMode == "multipart") { + return nil, errors.New("raw/multipart request DTOs may declare only path/form/header fields; body binding is manual") + } if op.Request != nil && (r.Method == "get" || r.Method == "head") { return nil, errors.New("GET/HEAD requests cannot declare JSON bodies; use form/path/header tags") } diff --git a/cmd/go-omni/apigen/contractMetadata.go b/cmd/go-omni/apigen/contractMetadata.go index b9de4dc..baac47b 100644 --- a/cmd/go-omni/apigen/contractMetadata.go +++ b/cmd/go-omni/apigen/contractMetadata.go @@ -140,6 +140,7 @@ func decorateDocument(doc map[string]any, spec *Spec) { } } op := doc["paths"].(map[string]any)[strings.Join(parts, "/")].(map[string]any)[r.Method].(map[string]any) + decorateRequestOptions(op, r) if r.ResponseFormat != "" { op["x-omni-response"] = r.ResponseFormat } diff --git a/cmd/go-omni/apigen/generate.go b/cmd/go-omni/apigen/generate.go index db3eb81..2ce2a2a 100644 --- a/cmd/go-omni/apigen/generate.go +++ b/cmd/go-omni/apigen/generate.go @@ -19,7 +19,7 @@ import ( ) const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" -const minimumVersion = "v0.8.0-preview" +const minimumVersion = "v0.9.0-preview" const generatedHeader = "// Code generated by go-omni api. DO NOT EDIT.\n" type Options struct { @@ -94,6 +94,16 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) { return Output{}, err } o.Dir = directory + for _, r := range c.Spec.Routes { + if !r.Manual { + path := filepath.Join(directory, "internal/handler", r.Group, lower(r.Handler)+"Manual.go") + if _, err := os.Lstat(path); err == nil { + return Output{}, fmt.Errorf("manual handler %s exists; migrate and remove it explicitly before switching to automatic generation", path) + } else if !errors.Is(err, os.ErrNotExist) { + return Output{}, err + } + } + } if err := resolveProfile(&o); err != nil { return Output{}, err } @@ -130,13 +140,13 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) { if !filepath.IsAbs(path) { path = filepath.Join(directory, path) } - if _, err := os.Stat(filepath.Join(path, "httpx", "response.go")); err == nil { + if _, err := os.Stat(filepath.Join(path, "contract", "scalar.go")); err == nil { localCurrent = true } } } if !localCurrent { - return Output{}, fmt.Errorf("existing framework dependency predates configurable responses; use %s or a current local replacement", minimumVersion) + return Output{}, fmt.Errorf("existing framework dependency predates migration extensions; use %s or a current local replacement", minimumVersion) } } if o.FrameworkDir != "" { @@ -309,6 +319,20 @@ func renderProject(o Options, c *Compiled, starter bool) (map[string][]byte, map put(generated, "internal/apierrors/errorsGen.go", generatedHeader+content) } types := "package types\n\n" + if len(c.Spec.Scalars) > 0 { + types += "import(" + for i, d := range c.Spec.Scalars { + types += fmt.Sprintf("scalar%d %q;", i, d.GoImport) + } + types += ")\n" + registry := fmt.Sprintf("package types\nimport %q\nfunc ContractScalars()[]contract.Scalar{return []contract.Scalar{", frameworkModule+"/contract") + for i, d := range c.Spec.Scalars { + types += fmt.Sprintf("type %s = scalar%d.%s\n", d.Name, i, d.GoType) + registry += fmt.Sprintf("contract.ScalarOf[%s](%q,%q,%t),", d.Name, d.JSONInput, d.JSONOutput, d.Text) + } + registry += "}}\n" + put(generated, "internal/types/scalarsGen.go", generatedHeader+registry) + } for _, t := range c.Spec.Types { types += "type " + t.Name + " struct {\n" for _, f := range t.Fields { @@ -331,6 +355,10 @@ func renderProject(o Options, c *Compiled, starter bool) (map[string][]byte, map put(generated, "internal/handler/responseGen.go", generatedHeader+policySource) imports := fmt.Sprintf("import (\"fmt\"; %q; %q; %q;", frameworkModule+"/httpx", frameworkModule+"/openapi", o.Module+"/internal/servicecontext") routes := "func RegisterRoutes(router openapi.Registrar, deps *servicecontext.Context) error {\n if router==nil || deps==nil {return fmt.Errorf(\"router and dependencies required\")}\n" + if len(c.Spec.Scalars) > 0 { + imports += fmt.Sprintf("%q;%q;", frameworkModule+"/contract", o.Module+"/internal/types") + routes += "if err:=contract.ValidateScalars(types.ContractScalars()...);err!=nil{return err}\n" + } middlewareSet := map[string]bool{} for _, r := range c.Spec.Routes { for _, m := range r.Middleware { @@ -347,7 +375,7 @@ func renderProject(o Options, c *Compiled, starter bool) (map[string][]byte, map } for i, group := range c.Groups { imports += fmt.Sprintf("h%d %q; s%d %q;", i, o.Module+"/internal/handler/"+group, i, o.Module+"/internal/service/"+group) - routes += fmt.Sprintf("h%d := h%d.New(s%d.New(deps))\n", i, i, i) + routes += fmt.Sprintf("h%d := h%d.New(s%d.New(deps),deps)\n", i, i, i) groupRoutes := []Route{} for _, r := range c.Spec.Routes { if r.Group == group { @@ -356,12 +384,16 @@ func renderProject(o Options, c *Compiled, starter bool) (map[string][]byte, map } packageName := filepath.Base(group) text := "package " + packageName + "\nimport(\"context\"; \"fmt\";" - needsTypes, needsBinder := false, false + needsTypes, needsBinder, hasAutomatic := false, false, false for _, r := range groupRoutes { + if r.Manual { + continue + } + hasAutomatic = true needsTypes = needsTypes || r.Request != "" || r.Response != "" - needsBinder = needsBinder || r.Request != "" + needsBinder = needsBinder || (r.Request != "" && !r.CustomBinding) } - text += fmt.Sprintf("%q;", frameworkModule+"/httpx") + text += fmt.Sprintf("%q;%q;", frameworkModule+"/httpx", o.Module+"/internal/servicecontext") if needsTypes { text += fmt.Sprintf("%q;", o.Module+"/internal/types") } @@ -371,18 +403,47 @@ func renderProject(o Options, c *Compiled, starter bool) (map[string][]byte, map text += ")\n" text += "type Service interface {\n" for _, r := range groupRoutes { - text += signature(r) + "\n" + if !r.Manual { + text += signature(r) + "\n" + } } - text += "}\ntype Handler struct {service Service}\nfunc New(s Service)*Handler {return &Handler{service:s}}\n" + text += "}\ntype Handler struct {service Service;deps *servicecontext.Context}\nfunc New(s Service,deps ...*servicecontext.Context)*Handler {h:=&Handler{service:s};if len(deps)>0{h.deps=deps[0]};return h}\n" for _, r := range groupRoutes { name := upper(r.Handler) - if r.Request != "" { - text += fmt.Sprintf("var %sRequest = contract.MustNew[types.%s]()\n", lower(r.Handler), r.Request) + // The response policy and body reader cap are installed before App middleware. + if r.MaxBodyBytes > 0 { + routes += fmt.Sprintf("httpx.HandleWithBodyLimit(router,%q,%q,%d,responseEncoder(%q,%t),h%d.%s", strings.ToUpper(r.Method), r.Path, r.MaxBodyBytes, responseName(r), r.ResponseFormat == "raw", i, name) + } else { + routes += fmt.Sprintf("httpx.HandleWithResponse(router,%q,%q,responseEncoder(%q,%t),h%d.%s", strings.ToUpper(r.Method), r.Path, responseName(r), r.ResponseFormat == "raw", i, name) + } + for _, m := range r.Middleware { + routes += fmt.Sprintf(",deps.Middlewares[%q]", m) + } + routes += ")\n" + if r.Manual { + put(seeds, "internal/handler/"+group+"/"+lower(r.Handler)+"Manual.go", fmt.Sprintf("package %s\nimport(%q;apierrors %q)\n// %s owns raw/multipart binding and response output. h.deps holds project dependencies.\n// This file is preserved on regeneration.\nfunc(h *Handler)%s(c *httpx.Context)error{return apierrors.New(501,\"NOT_IMPLEMENTED\",\"manual handler is not implemented\")}\n", packageName, frameworkModule+"/httpx", frameworkModule+"/errors", name, name)) + continue + } + if r.Request != "" && !r.CustomBinding { + scalarArgs := "" + if len(c.Spec.Scalars) > 0 { + scalarArgs = "types.ContractScalars()..." + } + text += fmt.Sprintf("var %sRequest = contract.MustNew[types.%s](%s)\n", lower(r.Handler), r.Request, scalarArgs) } text += fmt.Sprintf("func(h *Handler) %s(c *httpx.Context)error {\n", name) call := "c.Context()" if r.Request != "" { - text += fmt.Sprintf("var req types.%s\nif err:=%sRequest.Bind(c.Request(),httpx.RouteParams(c.Context()),&req);err!=nil{return err}\n", r.Request, lower(r.Handler)) + if r.CustomBinding { + text += fmt.Sprintf("var req types.%s\nif err:=h.bind%s(c,&req);err!=nil{return err}\n", r.Request, name) + put(seeds, "internal/handler/"+group+"/"+lower(r.Handler)+"Bind.go", fmt.Sprintf("package %s\nimport(%q;%q;apierrors %q)\n// bind%s is project-owned; repeated generation preserves it.\nfunc(h *Handler)bind%s(c *httpx.Context,req *types.%s)error{return apierrors.New(501,\"BINDING_NOT_IMPLEMENTED\",\"custom binding is not implemented\")}\n", packageName, frameworkModule+"/httpx", o.Module+"/internal/types", frameworkModule+"/errors", name, name, r.Request)) + } else { + limit := "" + if r.MaxBodyBytes > 0 { + limit = fmt.Sprintf(",%d", r.MaxBodyBytes) + } + text += fmt.Sprintf("var req types.%s\nif err:=%sRequest.Bind(c.Request(),httpx.RouteParams(c.Context()),&req%s);err!=nil{return err}\n", r.Request, lower(r.Handler), limit) + } call += ", &req" } if r.Response != "" { @@ -392,11 +453,6 @@ func renderProject(o Options, c *Compiled, starter bool) (map[string][]byte, map } else { text += fmt.Sprintf("if err:=h.service.%s(%s);err!=nil{return err};return c.NoContent()\n}\n", name, call) } - routes += fmt.Sprintf("httpx.HandleWithResponse(router,%q,%q,responseEncoder(%q,%t),h%d.%s", strings.ToUpper(r.Method), r.Path, responseName(r), r.ResponseFormat == "raw", i, name) - for _, m := range r.Middleware { - routes += fmt.Sprintf(",deps.Middlewares[%q]", m) - } - routes += ")\n" service := fmt.Sprintf("package %s\nimport(\"context\"; apierrors %q;", packageName, frameworkModule+"/errors") if r.Request != "" || r.Response != "" { service += fmt.Sprintf("%q;", o.Module+"/internal/types") @@ -417,6 +473,10 @@ func renderProject(o Options, c *Compiled, starter bool) (map[string][]byte, map } put(seeds, "internal/service/"+group+"/"+lower(r.Handler)+".go", service) } + if !hasAutomatic { + text = strings.Replace(text, strconv.Quote(frameworkModule+"/httpx")+";", "", 1) + text = strings.Replace(text, "\"context\";", "", 1) + } if !hasResponse(groupRoutes) { text = strings.Replace(text, "\"fmt\";", "", 1) } @@ -464,7 +524,7 @@ func renderProject(o Options, c *Compiled, starter bool) (map[string][]byte, map } func hasResponse(routes []Route) bool { for _, r := range routes { - if r.Response != "" { + if r.Response != "" && !r.Manual { return true } } diff --git a/cmd/go-omni/apigen/parse.go b/cmd/go-omni/apigen/parse.go index 515097e..118a5fa 100644 --- a/cmd/go-omni/apigen/parse.go +++ b/cmd/go-omni/apigen/parse.go @@ -18,6 +18,7 @@ type Spec struct { Errors []ErrorDefinition Auth []AuthDefinition Responses []ResponseDefinition + Scalars []ScalarDefinition Types []Type Routes []Route Sources map[string][]byte // relative to the entry API's directory @@ -30,6 +31,9 @@ type Type struct { } type Field struct{ Name, Type, Tag string } type Route struct { + Manual, CustomBinding bool + BodyMode, Produces string + MaxBodyBytes int64 Method, Path, Request, Response, Handler, Group, Summary string Middleware []string Auth string @@ -199,6 +203,10 @@ func (p *parser) parse() error { syntaxSeen := false for p.skip(); p.tokens[p.at].kind != scanner.EOF; p.skip() { switch p.peek() { + case "scalar": + if err := p.parseScalar(); err != nil { + return err + } case "syntax": if syntaxSeen { return p.err("duplicate syntax") @@ -393,7 +401,15 @@ func (p *parser) parseType() error { } expression := "" for p.peek() != "\n" && p.peek() != ";" && p.peek() != "}" && p.tokens[p.at].kind != scanner.RawString && p.tokens[p.at].kind != scanner.EOF { - expression += p.next().text + if p.peek() == "interface" { + p.next() + if p.expect("{") != nil || p.expect("}") != nil { + return p.err("only empty interface{} is supported; use any for dynamic JSON") + } + expression += "any" + } else { + expression += p.next().text + } } if expression == "" { return p.err("field type required; embedded types are not supported") @@ -453,6 +469,28 @@ func (p *parser) parseService(server map[string]string) error { } seen[annotation] = true switch annotation { + case "handlerMode": + mode, err := p.ident() + if err != nil { + return err + } + if mode != "manual" { + return p.err("handlerMode must be manual") + } + route.Manual = true + case "binding": + mode, err := p.ident() + if err != nil { + return err + } + if mode != "custom" { + return p.err("binding must be custom") + } + route.CustomBinding = true + case "produces": + route.Produces, e = p.quoted() + case "body": + e = p.parseBody(&route) case "handler": route.Handler, e = p.ident() case "doc": @@ -512,7 +550,7 @@ func (p *parser) parseService(server map[string]string) error { if p.peek() != "\n" && p.peek() != ";" && p.peek() != "}" { return p.err("unexpected route suffix") } - if route.Response == "" && !seen["status"] { + if route.Response == "" && !seen["status"] && !route.Manual { route.Status = 204 } p.spec.Routes = append(p.spec.Routes, route) diff --git a/cmd/go-omni/apigen/requestOptions.go b/cmd/go-omni/apigen/requestOptions.go new file mode 100644 index 0000000..0109b75 --- /dev/null +++ b/cmd/go-omni/apigen/requestOptions.go @@ -0,0 +1,95 @@ +package apigen + +import ( + "errors" + "fmt" + "git.yuntuo.hk/k2safe/go-omni/contract" + "mime" + "strconv" + "strings" +) + +func (p *parser) parseBody(r *Route) error { + props, err := p.metadata() + if err != nil { + return err + } + for key, value := range props { + switch key { + case "mode": + r.BodyMode = value + case "maxBytes": + r.MaxBodyBytes, err = strconv.ParseInt(value, 10, 64) + if err != nil || r.MaxBodyBytes <= 0 || r.MaxBodyBytes > 1<<40 { + return p.err("maxBytes must be 1..1099511627776 bytes") + } + default: + return fmt.Errorf("unknown body property %s", key) + } + } + if r.MaxBodyBytes == 0 { + r.MaxBodyBytes = contract.DefaultMaxBodyBytes + } + return nil +} +func validateRouteExtensions(r Route) error { + if r.BodyMode != "" && r.BodyMode != "json" && r.BodyMode != "raw" && r.BodyMode != "multipart" { + return errors.New("body mode must be json, raw or multipart") + } + if (r.BodyMode == "raw" || r.BodyMode == "multipart") && !r.Manual { + return errors.New("raw/multipart body requires @handlerMode manual") + } + if r.CustomBinding && (r.Manual || r.Request == "") { + return errors.New("@binding custom requires an automatic handler with a request DTO") + } + if r.MaxBodyBytes > 0 && (r.Method == "get" || r.Method == "head") { + return errors.New("GET/HEAD cannot declare a body limit") + } + if r.Produces != "" { + if !r.Manual { + return errors.New("@produces requires @handlerMode manual") + } + media, params, err := mime.ParseMediaType(r.Produces) + if err != nil || len(params) != 0 || media != r.Produces || strings.Contains(media, "*") { + return errors.New("produces requires a concrete media type without parameters") + } + if r.Response != "" || r.Status == 204 || r.Method == "head" { + return errors.New("@produces uses a manual response without a DTO, 204 or HEAD") + } + } + if r.Manual && r.Status == 204 && r.Response != "" { + return errors.New("204 cannot declare a response") + } + return nil +} + +func decorateRequestOptions(op map[string]any, r Route) { + if r.Manual { + op["x-omni-handler-mode"] = "manual" + } + if r.CustomBinding { + op["x-omni-binding"] = "custom" + } + if r.MaxBodyBytes > 0 { + op["x-omni-max-body-bytes"] = r.MaxBodyBytes + } + if r.BodyMode == "raw" || r.BodyMode == "multipart" { + media := "application/octet-stream" + schema := map[string]any{"type": "string", "format": "binary"} + if r.BodyMode == "multipart" { + media = "multipart/form-data" + schema = map[string]any{"type": "object", "additionalProperties": true} + } + op["requestBody"] = map[string]any{"required": true, "content": map[string]any{media: map[string]any{"schema": schema}}} + } + if r.Produces != "" { + schema := map[string]any{"type": "string"} + if r.Produces == "application/octet-stream" { + schema["format"] = "binary" + } + if r.Produces == "application/json" { + schema = map[string]any{} + } + op["responses"].(map[string]any)[strconv.Itoa(r.Status)] = map[string]any{"description": "Success", "content": map[string]any{r.Produces: map[string]any{"schema": schema}}} + } +} diff --git a/cmd/go-omni/apigen/scalar.go b/cmd/go-omni/apigen/scalar.go new file mode 100644 index 0000000..ebe0cfc --- /dev/null +++ b/cmd/go-omni/apigen/scalar.go @@ -0,0 +1,74 @@ +package apigen + +import ( + "fmt" + "git.yuntuo.hk/k2safe/go-omni/contract" + "golang.org/x/mod/module" + "reflect" + "strings" +) + +type ScalarDefinition struct { + Name, GoImport, GoType, JSONInput, JSONOutput, Position string + Text bool +} + +func (p *parser) parseScalar() error { + pos := p.next().position + name, err := p.ident() + if err != nil { + return err + } + p.skip() + props, err := p.metadata() + if err != nil { + return err + } + d := ScalarDefinition{Name: name, Position: pos} + for key, value := range props { + switch key { + case "goImport": + d.GoImport = value + case "goType": + d.GoType = value + case "jsonInput": + d.JSONInput = value + case "jsonOutput": + d.JSONOutput = value + case "text": + if value != "true" && value != "false" { + return p.err("text must be true or false") + } + d.Text = value == "true" + default: + return fmt.Errorf("%s: unknown scalar property %s", pos, key) + } + } + p.spec.Scalars = append(p.spec.Scalars, d) + return nil +} +func compileScalars(spec *Spec, types map[string]Type) (map[string]reflect.Type, []contract.Scalar, error) { + out := map[string]reflect.Type{} + var defs []contract.Scalar + goTypes := map[string]bool{} + for _, d := range spec.Scalars { + key := d.GoImport + "." + d.GoType + if d.Name == "ContractScalars" || !exportedName.MatchString(d.Name) || !exportedName.MatchString(d.GoType) || out[d.Name] != nil || types[d.Name].Name != "" || goTypes[key] { + return nil, nil, fmt.Errorf("%s: scalar names and Go types must be unique; names must be UpperCamelCase", d.Position) + } + if err := module.CheckImportPath(d.GoImport); err != nil || strings.Contains(d.GoImport, "\\") { + return nil, nil, fmt.Errorf("%s: invalid scalar goImport", d.Position) + } + // Unique stand-in types allow CLI schema checks without executing project code. + t := reflect.StructOf([]reflect.StructField{{Name: d.Name, Type: reflect.TypeOf(false)}}) + def := contract.Scalar{Type: t, JSONInput: d.JSONInput, JSONOutput: d.JSONOutput, Text: d.Text} + probe := reflect.StructOf([]reflect.StructField{{Name: "Value", Type: t, Tag: `json:"value"`}}) + if _, err := contract.Describe(probe, def); err != nil { + return nil, nil, fmt.Errorf("%s: %w", d.Position, err) + } + out[d.Name] = t + defs = append(defs, def) + goTypes[key] = true + } + return out, defs, nil +} diff --git a/cmd/go-omni/apigen/testdata/compatibility/consumer_test.go b/cmd/go-omni/apigen/testdata/compatibility/consumer_test.go new file mode 100644 index 0000000..273c752 --- /dev/null +++ b/cmd/go-omni/apigen/testdata/compatibility/consumer_test.go @@ -0,0 +1,117 @@ +package compatibility_test + +import ( + "bytes" + "context" + "crypto/hmac" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "example.com/compatibility/internal/handler" + "example.com/compatibility/internal/servicecontext" + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/logx" + "io" + "mime/multipart" + "net/http/httptest" + "os" + "strings" + "testing" +) + +func TestGeneratedCompatibility(t *testing.T) { + logger, err := logx.New(logx.Config{Writer: io.Discard}) + if err != nil { + t.Fatal(err) + } + defer logger.Close() + deps, err := servicecontext.New(context.Background(), logger) + if err != nil { + t.Fatal(err) + } + defer deps.Close() + app := omni.New() + defer app.Close() + if err = handler.RegisterRoutes(app, deps); err != nil { + t.Fatal(err) + } + send := func(method, path, body, contentType string) *httptest.ResponseRecorder { + r := httptest.NewRequest(method, path, strings.NewReader(body)) + r.Header.Set("Content-Type", contentType) + w := httptest.NewRecorder() + app.ServeHTTP(w, r) + return w + } + for _, id := range []string{`9223372036854775807`, `"9223372036854775807"`} { + w := send("POST", "/echo", `{"id":`+id+`,"data":{"n":9223372036854775807}}`, "application/json") + if w.Code != 200 || !strings.Contains(w.Body.String(), `"id":"9223372036854775807"`) || !strings.Contains(w.Body.String(), `"n":9223372036854775807`) { + t.Fatalf("scalar/dynamic: %d %s", w.Code, w.Body) + } + } + for _, body := range []string{`{"id":true,"data":null}`, `{"id":9223372036854775808,"data":null}`, `{"id":1,"data":{"x":1,"x":2}}`, `{"id":1,"data":null,"extra":1}`} { + if w := send("POST", "/echo", body, "application/json"); w.Code != 400 { + t.Fatalf("accepted %s: %d", body, w.Code) + } + } + payload := `{"id":1,"data":"` + strings.Repeat("x", (1<<20)+1) + `"}` + if w := send("POST", "/echo", payload, "application/json"); w.Code != 200 { + t.Fatalf("route limit not passed to Binder: %d", w.Code) + } + if w := send("POST", "/echo", `{"id":1,"data":"`+strings.Repeat("x", 2<<20)+`"}`, "application/json"); w.Code != 413 { + t.Fatalf("large JSON: %d", w.Code) + } + if w := send("GET", "/items/9223372036854775807", "", ""); w.Code != 200 || !strings.Contains(w.Body.String(), `"id":"9223372036854775807"`) { + t.Fatalf("text scalar: %d %s", w.Code, w.Body) + } + if w := send("POST", "/custom", `{"value":"hello"}`, "application/json"); w.Code != 200 || !strings.Contains(w.Body.String(), "HELLO") { + t.Fatalf("custom binder: %d %s", w.Code, w.Body) + } + raw := `{ "event": 1, "id": "original bytes" }` + mac := hmac.New(sha256.New, []byte("fixture-only-key")) + mac.Write([]byte(raw)) + r := httptest.NewRequest("POST", "/callback", strings.NewReader(raw)) + r.Header.Set("X-Signature", hex.EncodeToString(mac.Sum(nil))) + w := httptest.NewRecorder() + app.ServeHTTP(w, r) + if w.Code != 200 || w.Body.String() != "accepted" { + t.Fatalf("raw callback: %d %s", w.Code, w.Body) + } + r = httptest.NewRequest("POST", "/callback", strings.NewReader(strings.Repeat("x", 129))) + r.ContentLength = -1 + w = httptest.NewRecorder() + app.ServeHTTP(w, r) + if w.Code != 413 { + t.Fatalf("chunked body limit: %d %s", w.Code, w.Body) + } + temp := t.TempDir() + t.Setenv("TMPDIR", temp) + var form bytes.Buffer + writer := multipart.NewWriter(&form) + part, err := writer.CreateFormFile("file", "sample.bin") + if err != nil { + t.Fatal(err) + } + part.Write(bytes.Repeat([]byte("x"), 8192)) + writer.Close() + w = send("POST", "/upload", form.String(), writer.FormDataContentType()) + if w.Code != 200 || !strings.Contains(w.Body.String(), `"bytes":8192`) { + t.Fatalf("upload: %d %s", w.Code, w.Body) + } + entries, err := os.ReadDir(temp) + if err != nil || len(entries) != 0 { + t.Fatalf("multipart temp files leaked: %v %v", entries, err) + } + w = send("GET", "/download", "", "") + if w.Code != 200 || w.Body.Len() != 128<<10 || w.Header().Get("Content-Disposition") == "" { + t.Fatalf("download wrapped: %d %d", w.Code, w.Body.Len()) + } + var doc map[string]any + if err = json.Unmarshal([]byte(handler.OpenAPIDocument), &doc); err != nil { + t.Fatal(err) + } + paths := doc["paths"].(map[string]any) + media := paths["/download"].(map[string]any)["get"].(map[string]any)["responses"].(map[string]any)["200"].(map[string]any)["content"].(map[string]any) + if media["application/octet-stream"] == nil || media["application/json"] != nil { + t.Fatal("download schema is JSON") + } +} diff --git a/cmd/go-omni/apigen/testdata/compatibility/download.go b/cmd/go-omni/apigen/testdata/compatibility/download.go new file mode 100644 index 0000000..dd0a815 --- /dev/null +++ b/cmd/go-omni/apigen/testdata/compatibility/download.go @@ -0,0 +1,16 @@ +package files + +import ( + "git.yuntuo.hk/k2safe/go-omni/httpx" + "io" + "net/http" + "strings" +) + +func (h *Handler) Download(c *httpx.Context) error { + return httpx.WrapHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/octet-stream") + w.Header().Set("Content-Disposition", `attachment; filename="demo.bin"`) + _, _ = io.Copy(w, strings.NewReader(strings.Repeat("x", 128<<10))) + }))(c) +} diff --git a/cmd/go-omni/apigen/testdata/compatibility/echo.go b/cmd/go-omni/apigen/testdata/compatibility/echo.go new file mode 100644 index 0000000..0f2944e --- /dev/null +++ b/cmd/go-omni/apigen/testdata/compatibility/echo.go @@ -0,0 +1,16 @@ +package demo + +import ( + "context" + "example.com/compatibility/internal/types" +) + +func (s *Service) Echo(ctx context.Context, req *types.EchoReq) (*types.EchoResp, error) { + return &types.EchoResp{ID: req.ID, Data: req.Data}, nil +} +func (s *Service) Item(ctx context.Context, req *types.ItemReq) (*types.EchoResp, error) { + return &types.EchoResp{ID: req.ID, Data: map[string]any{"found": true}}, nil +} +func (s *Service) Custom(ctx context.Context, req *types.CustomReq) (*types.CustomResp, error) { + return &types.CustomResp{Value: req.Value}, nil +} diff --git a/cmd/go-omni/apigen/testdata/compatibility/hooks.go b/cmd/go-omni/apigen/testdata/compatibility/hooks.go new file mode 100644 index 0000000..12e5977 --- /dev/null +++ b/cmd/go-omni/apigen/testdata/compatibility/hooks.go @@ -0,0 +1,58 @@ +package demo + +import ( + "crypto/hmac" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "example.com/compatibility/internal/types" + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "io" + "net/http" + "strings" +) + +func (h *Handler) bindCustom(c *httpx.Context, req *types.CustomReq) error { + decoder := json.NewDecoder(c.Request().Body) + decoder.DisallowUnknownFields() + if err := decoder.Decode(req); err != nil { + return apierrors.BadRequest("INVALID_REQUEST", "invalid request").WithCause(err) + } + var extra any + if err := decoder.Decode(&extra); err != io.EOF { + return apierrors.BadRequest("INVALID_REQUEST", "one JSON value required").WithCause(err) + } + req.Value = strings.ToUpper(req.Value) + return nil +} +func (h *Handler) Upload(c *httpx.Context) error { + // The small memory threshold deliberately exercises disk spill and cleanup. + if err := c.Request().ParseMultipartForm(8); err != nil { + return err + } + file, _, err := c.Request().FormFile("file") + if err != nil { + return err + } + defer file.Close() + count, err := io.Copy(io.Discard, file) + if err != nil { + return err + } + return c.OK(map[string]any{"bytes": count}) +} +func (h *Handler) Callback(c *httpx.Context) error { + raw, err := io.ReadAll(c.Request().Body) + if err != nil { + return err + } + // This fixed key belongs only to the test fixture. Applications inject secrets. + mac := hmac.New(sha256.New, []byte("fixture-only-key")) + mac.Write(raw) + signature, err := hex.DecodeString(c.Header("X-Signature")) + if err != nil || !hmac.Equal(signature, mac.Sum(nil)) { + return c.Text(http.StatusUnauthorized, "invalid signature") + } + return c.Text(http.StatusOK, "accepted") +} diff --git a/cmd/go-omni/apigen/write.go b/cmd/go-omni/apigen/write.go index 767fbbc..e706d14 100644 --- a/cmd/go-omni/apigen/write.go +++ b/cmd/go-omni/apigen/write.go @@ -252,7 +252,7 @@ func apply(root, module, entry string, generated, seeds map[string][]byte, check } func digest(data []byte) string { sum := sha256.Sum256(data); return hex.EncodeToString(sum[:]) } func ownedPath(name string) bool { - if name == "internal/handler/responseGen.go" { + if name == "internal/handler/responseGen.go" || name == "internal/types/scalarsGen.go" { return true } if name == "internal/apierrors/errorsGen.go" || name == "docs/openapi.json" || name == "internal/types/typesGen.go" || name == "internal/handler/routesGen.go" || name == "internal/handler/openapiGen.go" || name == "internal/middleware/registryGen.go" { diff --git a/cmd/go-omni/scaffold/version.go b/cmd/go-omni/scaffold/version.go index fd1db8b..67da8b3 100644 --- a/cmd/go-omni/scaffold/version.go +++ b/cmd/go-omni/scaffold/version.go @@ -10,7 +10,7 @@ const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" // Source builds can carry an unpublished VCS version (including +dirty). // Keep this fallback in sync with the next release; installed CLIs use their // downloaded module version, which also has a module checksum. -const defaultFrameworkVersion = "v0.8.0-preview" +const defaultFrameworkVersion = "v0.9.0-preview" // This target is not a claim that the release already exists. Source users // select a local framework checkout until the module templates are published. diff --git a/contract/bind.go b/contract/bind.go index a44f67e..547c944 100644 --- a/contract/bind.go +++ b/contract/bind.go @@ -19,7 +19,7 @@ import ( // strictly decoded, and limited to 1 MiB unless maxBytes is supplied. Query // parameters never overwrite path/body/header values. form means URL query only. func (b *Binder) Bind(r *http.Request, params map[string]string, target any, maxBytes ...int64) error { - if b == nil || r == nil { + if b == nil || r == nil || b.schemaOnly { return apierrors.Internal(errors.New("contract: nil binder or request")) } v := reflect.ValueOf(target) @@ -65,6 +65,10 @@ func (b *Binder) Bind(r *http.Request, params map[string]string, target any, max return tooLarge() } if err != io.EOF { + var maxErr *http.MaxBytesError + if errors.As(err, &maxErr) { + return tooLarge() + } return invalid(errors.New("exactly one JSON value required")) } var ok bool @@ -126,10 +130,16 @@ func tooLarge() error { return apierrors.New(413, "BODY_TOO_LARGE", "request bod func fromStrings(n *node, values []string) (any, error) { n = base(n) + if n.scalar != nil { + if len(values) != 1 { + return nil, errors.New("duplicate scalar parameter") + } + return scalarText(values[0]), nil + } if n.typ.Kind() == reflect.Slice { out := make([]any, 0, len(values)) for _, s := range values { - v, err := parseScalar(s, base(n.elem).typ) + v, err := fromStrings(n.elem, []string{s}) if err != nil { return nil, err } @@ -158,6 +168,17 @@ func assignField(f field, dst reflect.Value, value any, present bool) error { return checkField(f, dst) } func setValue(n *node, dst reflect.Value, value any) error { + if n.scalar != nil { + return n.scalar.decode(dst, value) + } + if n.typ.Kind() == reflect.Interface && n.typ.NumMethod() == 0 { + if value == nil { + dst.SetZero() + } else { + dst.Set(reflect.ValueOf(value)) + } + return nil + } if value == nil { if n.typ.Kind() == reflect.Pointer { dst.SetZero() diff --git a/contract/contract.go b/contract/contract.go index 89d0488..d7bb7ef 100644 --- a/contract/contract.go +++ b/contract/contract.go @@ -23,10 +23,12 @@ var valuesValidator = validator.New() // Binder is immutable after compilation and safe for concurrent requests. type Binder struct { - root *node - typ reflect.Type + root *node + typ reflect.Type + schemaOnly bool } type node struct { + scalar *Scalar typ reflect.Type schema *openapi.Schema fields []field @@ -42,9 +44,11 @@ type field struct { binding string } -func New[T any]() (*Binder, error) { return ForType(reflect.TypeOf((*T)(nil)).Elem()) } -func MustNew[T any]() *Binder { - b, err := New[T]() +func New[T any](scalars ...Scalar) (*Binder, error) { + return ForType(reflect.TypeOf((*T)(nil)).Elem(), scalars...) +} +func MustNew[T any](scalars ...Scalar) *Binder { + b, err := New[T](scalars...) if err != nil { panic(err) } @@ -52,20 +56,36 @@ func MustNew[T any]() *Binder { } // ForType also lets generators check the exact runtime contract before writing files. -// The root must be a struct; recursive and custom-marshaling types are rejected. -func ForType(t reflect.Type) (*Binder, error) { +// The root must be a struct; custom marshalers require explicit scalar registration. +func ForType(t reflect.Type, scalars ...Scalar) (*Binder, error) { return build(t, false, scalars) } + +// Describe compiles schemas without loading or executing application Go types. +// The returned Binder cannot bind requests; it is for generators only. +func Describe(t reflect.Type, scalars ...Scalar) (*Binder, error) { return build(t, true, scalars) } + +func build(t reflect.Type, schemaOnly bool, scalars []Scalar) (*Binder, error) { if t == nil || t.Kind() != reflect.Struct { return nil, errors.New("contract: root must be a struct") } + c := compiler{scalars: map[reflect.Type]Scalar{}} + for _, scalar := range scalars { + if err := scalar.validate(schemaOnly); err != nil { + return nil, err + } + if _, exists := c.scalars[scalar.Type]; exists { + return nil, fmt.Errorf("contract: duplicate scalar type %s", scalar.Type) + } + c.scalars[scalar.Type] = scalar + } budget := 10000 - n, err := compile(t, true, map[reflect.Type]bool{}, 0, &budget) + n, err := c.compile(t, true, map[reflect.Type]bool{}, 0, &budget) if err != nil { return nil, err } - return &Binder{root: n, typ: t}, nil + return &Binder{root: n, typ: t, schemaOnly: schemaOnly}, nil } -func compile(t reflect.Type, root bool, stack map[reflect.Type]bool, depth int, budget *int) (*node, error) { +func (c compiler) compile(t reflect.Type, root bool, stack map[reflect.Type]bool, depth int, budget *int) (*node, error) { *budget-- if *budget < 0 { return nil, errors.New("contract: schema expansion exceeds 10000 nodes") @@ -79,21 +99,32 @@ func compile(t reflect.Type, root bool, stack map[reflect.Type]bool, depth int, stack[t] = true defer delete(stack, t) n := &node{typ: t, schema: &openapi.Schema{}} + if scalar, ok := c.scalars[t]; ok { + if root { + return nil, errors.New("contract: root cannot be a scalar") + } + n.scalar = &scalar + n.schema = scalar.inputSchema() + return n, nil + } // Wire behavior must be described completely by these tags. marshaler := reflect.TypeOf((*json.Marshaler)(nil)).Elem() unmarshaler := reflect.TypeOf((*json.Unmarshaler)(nil)).Elem() - if t.Implements(marshaler) || reflect.PointerTo(t).Implements(marshaler) || t.Implements(unmarshaler) || reflect.PointerTo(t).Implements(unmarshaler) { - return nil, errors.New("contract: custom JSON marshalers are unsupported") + if t.Kind() != reflect.Pointer && (t.Implements(marshaler) || reflect.PointerTo(t).Implements(marshaler) || t.Implements(unmarshaler) || reflect.PointerTo(t).Implements(unmarshaler)) { + return nil, errors.New("contract: custom JSON marshalers require explicit Scalar registration") } switch t.Kind() { + case reflect.Interface: + if t.NumMethod() != 0 { + return nil, errors.New("contract: only any / interface{} is supported") + } case reflect.Pointer: - e, err := compile(t.Elem(), false, stack, depth+1, budget) + e, err := c.compile(t.Elem(), false, stack, depth+1, budget) if err != nil { return nil, err } n.elem = e - *n.schema = *e.schema - n.schema.Nullable = true + n.schema = nullableSchema(e.schema) case reflect.String: n.schema.Type = "string" case reflect.Bool: @@ -132,10 +163,10 @@ func compile(t reflect.Type, root bool, stack map[reflect.Type]bool, depth int, if t.Kind() == reflect.Map && t.Key().Kind() != reflect.String { return nil, errors.New("contract: map keys must be strings") } - if t.Kind() == reflect.Slice && t.Elem().Kind() == reflect.Uint8 { + if t.Kind() == reflect.Slice && t.Elem().Kind() == reflect.Uint8 && c.scalars[t.Elem()].Type == nil { return nil, errors.New("contract: byte slices use base64 in Go; use string for binary data") } - e, err := compile(t.Elem(), false, stack, depth+1, budget) + e, err := c.compile(t.Elem(), false, stack, depth+1, budget) if err != nil { return nil, err } @@ -158,7 +189,7 @@ func compile(t reflect.Type, root bool, stack map[reflect.Type]bool, depth int, if sf.PkgPath != "" || sf.Anonymous { return nil, fmt.Errorf("contract: %s must be an exported named field", sf.Name) } - f, err := compileField(sf, root, stack, depth+1, budget) + f, err := c.compileField(sf, root, stack, depth+1, budget) if err != nil { return nil, fmt.Errorf("contract: %s: %w", sf.Name, err) } @@ -182,7 +213,7 @@ func compile(t reflect.Type, root bool, stack map[reflect.Type]bool, depth int, return n, nil } -func compileField(sf reflect.StructField, root bool, stack map[reflect.Type]bool, depth int, budget *int) (field, error) { +func (c compiler) compileField(sf reflect.StructField, root bool, stack map[reflect.Type]bool, depth int, budget *int) (field, error) { f := field{} // Reject typos and unsupported tags instead of silently losing validation. tags, err := ParseTags(string(sf.Tag)) @@ -218,11 +249,37 @@ func compileField(sf reflect.StructField, root bool, stack map[reflect.Type]bool if (f.source == "header" && !headerName.MatchString(f.name)) || (f.source != "header" && !wireName.MatchString(f.name)) { return f, errors.New("wire names must be lowerCamelCase (headers use HTTP names)") } - n, err := compile(sf.Type, false, stack, depth, budget) + n, err := c.compile(sf.Type, false, stack, depth, budget) if err != nil { return f, err } f.value = n + opaque := base(n).scalar + isList := base(n).typ.Kind() == reflect.Slice && opaque == nil + if isList { + opaque = base(base(n).elem).scalar + } + if opaque != nil { + if isList && (f.source == "path" || f.source == "header") { + return f, errors.New("path/header values must be scalars") + } + if f.source != "json" && !opaque.Text { + return f, errors.New("custom scalar needs text: true and UnmarshalText for path/query/header") + } + if f.binding != "" { + return f, errors.New("custom scalar constraints belong in its decoder; use optional/pointer for presence and null") + } + for _, option := range parts[1:] { + if option != "optional" && option != "omitempty" { + return f, errors.New("custom scalar supports only optional/omitempty field options") + } + f.optional = true + } + if f.source == "path" && (f.optional || sf.Type.Kind() == reflect.Pointer) { + return f, errors.New("path fields must be required non-pointer values") + } + return f, nil + } scalar := base(n).typ.Kind() if f.source != "json" && !isScalar(scalar) && !(scalar == reflect.Slice && isScalar(base(base(n).elem).typ.Kind())) { return f, errors.New("path/query/header values must be scalars or scalar slices") @@ -370,7 +427,7 @@ func (b *Binder) ResponseSchema() (*openapi.Schema, error) { return nil, errors.New("contract: response fields must use json tags") } } - return copySchema(b.root.schema), nil + return responseSchema(b.root), nil } func (b *Binder) Parameters() []openapi.Parameter { var out []openapi.Parameter @@ -382,7 +439,13 @@ func (b *Binder) Parameters() []openapi.Parameter { if in == "form" { in = "query" } - out = append(out, openapi.Parameter{Name: f.name, In: in, Required: !f.optional && !f.hasDefault, Schema: copySchema(f.value.schema)}) + schema := copySchema(f.value.schema) + if base(f.value).scalar != nil { + schema = &openapi.Schema{Type: "string"} + } else if base(f.value).typ.Kind() == reflect.Slice && base(base(f.value).elem).scalar != nil { + schema.Items = &openapi.Schema{Type: "string"} + } + out = append(out, openapi.Parameter{Name: f.name, In: in, Required: !f.optional && !f.hasDefault, Schema: schema}) } return out } diff --git a/contract/scalar.go b/contract/scalar.go new file mode 100644 index 0000000..13d47a5 --- /dev/null +++ b/contract/scalar.go @@ -0,0 +1,199 @@ +package contract + +import ( + "encoding" + "encoding/json" + "errors" + "fmt" + "reflect" + "strconv" + "strings" + + "git.yuntuo.hk/k2safe/go-omni/openapi" +) + +// Scalar explicitly opts one application Go type into custom wire conversion. +// Input is a | separated union of scalar wire types; Output is one wire type. +// The value must implement json.Marshaler, its pointer json.Unmarshaler, and +// Text=true additionally requires encoding.TextUnmarshaler on the pointer. +// No process-global registry or mutation of other binders is involved. +type Scalar struct { + Type reflect.Type + JSONInput, JSONOutput string + Text bool +} + +func ScalarOf[T any](input, output string, text bool) Scalar { + return Scalar{Type: reflect.TypeOf((*T)(nil)).Elem(), JSONInput: input, JSONOutput: output, Text: text} +} + +type compiler struct { + scalars map[reflect.Type]Scalar +} +type scalarText string + +func (s Scalar) validate(schemaOnly bool) error { + if s.Type == nil || s.Type.Kind() == reflect.Pointer || s.Type.Kind() == reflect.Interface { + return errors.New("contract: scalar requires a concrete non-pointer type") + } + seen := map[string]bool{} + for _, kind := range strings.Split(s.JSONInput, "|") { + kind = strings.TrimSpace(kind) + if wireSchema(kind) == nil || seen[kind] { + return fmt.Errorf("contract: invalid/duplicate scalar input %q", kind) + } + seen[kind] = true + } + if wireSchema(s.JSONOutput) == nil { + return fmt.Errorf("contract: invalid scalar output %q", s.JSONOutput) + } + if !schemaOnly { + if !s.Type.Implements(reflect.TypeOf((*json.Marshaler)(nil)).Elem()) || !reflect.PointerTo(s.Type).Implements(reflect.TypeOf((*json.Unmarshaler)(nil)).Elem()) { + return fmt.Errorf("contract: %s needs value MarshalJSON and pointer UnmarshalJSON", s.Type) + } + if s.Text && !reflect.PointerTo(s.Type).Implements(reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem()) { + return fmt.Errorf("contract: %s needs UnmarshalText", s.Type) + } + } + return nil +} +func wireSchema(kind string) *openapi.Schema { + switch kind { + case "string": + return &openapi.Schema{Type: "string"} + case "bool": + return &openapi.Schema{Type: "boolean"} + case "int32", "int64": + return &openapi.Schema{Type: "integer", Format: kind} + case "float64": + return &openapi.Schema{Type: "number", Format: "double"} + } + return nil +} +func (s Scalar) inputSchema() *openapi.Schema { + var alternatives []*openapi.Schema + for _, kind := range strings.Split(s.JSONInput, "|") { + alternatives = append(alternatives, wireSchema(strings.TrimSpace(kind))) + } + if len(alternatives) == 1 { + return alternatives[0] + } + return &openapi.Schema{AnyOf: alternatives} +} +func (s Scalar) decode(dst reflect.Value, value any) (err error) { + // User decoders may fail; do not allow their panic to escape a binder. + defer func() { + if r := recover(); r != nil { + err = errors.New("custom scalar decoder panicked") + } + }() + if text, ok := value.(scalarText); ok { + decoder, ok := dst.Addr().Interface().(encoding.TextUnmarshaler) + if !s.Text || !ok { + return errors.New("text decoder not available") + } + return decoder.UnmarshalText([]byte(text)) + } + allowed := false + for _, kind := range strings.Split(s.JSONInput, "|") { + if scalarAccepts(strings.TrimSpace(kind), value) { + allowed = true + break + } + } + if !allowed { + return errors.New("value does not match declared scalar input") + } + decoder, ok := dst.Addr().Interface().(json.Unmarshaler) + if !ok { + return errors.New("JSON decoder not available") + } + raw, err := json.Marshal(value) // json.Number preserves exact integer tokens. + if err != nil { + return err + } + return decoder.UnmarshalJSON(raw) +} +func scalarAccepts(kind string, value any) bool { + switch kind { + case "string": + _, ok := value.(string) + return ok + case "bool": + _, ok := value.(bool) + return ok + case "int32", "int64": + n, ok := value.(json.Number) + if !ok { + return false + } + bits := 64 + if kind == "int32" { + bits = 32 + } + _, err := strconv.ParseInt(string(n), 10, bits) + return err == nil + case "float64": + n, ok := value.(json.Number) + if !ok { + return false + } + _, err := strconv.ParseFloat(string(n), 64) + return err == nil + } + return false +} +func responseSchema(n *node) *openapi.Schema { + b := base(n) + if b.scalar != nil { + s := wireSchema(b.scalar.JSONOutput) + if n.typ.Kind() == reflect.Pointer { + s = nullableSchema(s) + } + return s + } + // Preserve the field's own constraints (including pointer range/enum and + // collection length) while changing only nested custom scalar wire types. + s := copySchema(n.schema) + if b.typ.Kind() == reflect.Slice { + s.Items = responseSchema(b.elem) + } + if b.typ.Kind() == reflect.Map { + s.AdditionalProperties = responseSchema(b.elem) + } + for _, f := range b.fields { + if f.source == "json" { + s.Properties[f.name] = responseSchema(f.value) + } + } + return s +} + +// ValidateScalars checks the application's explicitly registered method contracts. +func ValidateScalars(scalars ...Scalar) error { + seen := map[reflect.Type]bool{} + for _, s := range scalars { + if err := s.validate(false); err != nil { + return err + } + if seen[s.Type] { + return fmt.Errorf("contract: duplicate scalar type %s", s.Type) + } + seen[s.Type] = true + } + return nil +} + +// OpenAPI 3.0 nullable only applies alongside an explicit type. A union needs +// an explicit null alternative; an unconstrained dynamic schema already accepts null. +func nullableSchema(schema *openapi.Schema) *openapi.Schema { + s := copySchema(schema) + if s.Type != "" { + s.Nullable = true + return s + } + if len(s.AnyOf) > 0 { + s.AnyOf = append(s.AnyOf, &openapi.Schema{Type: "string", Nullable: true, Enum: []any{nil}}) + } + return s +} diff --git a/contract/scalar_test.go b/contract/scalar_test.go new file mode 100644 index 0000000..c0e37a7 --- /dev/null +++ b/contract/scalar_test.go @@ -0,0 +1,222 @@ +package contract + +import ( + "encoding/json" + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" + "net/http" + "net/http/httptest" + "reflect" + "strconv" + "strings" + "testing" +) + +type flexibleID int64 + +func (v flexibleID) MarshalJSON() ([]byte, error) { + return json.Marshal(strconv.FormatInt(int64(v), 10)) +} +func (v *flexibleID) UnmarshalJSON(b []byte) error { + var s string + if len(b) > 0 && b[0] == '"' { + if err := json.Unmarshal(b, &s); err != nil { + return err + } + } else { + s = string(b) + } + return v.UnmarshalText([]byte(s)) +} +func (v *flexibleID) UnmarshalText(b []byte) error { + n, err := strconv.ParseInt(string(b), 10, 64) + if err == nil { + *v = flexibleID(n) + } + return err +} +func TestDynamicJSONAndCustomScalars(t *testing.T) { + type request struct { + ID flexibleID `json:"id"` + Dynamic any `json:"dynamic"` + IDs []flexibleID `json:"ids,optional"` + } + scalar := ScalarOf[flexibleID]("string|int64", "string", true) + b, err := New[request](scalar) + if err != nil { + t.Fatal(err) + } + for _, id := range []string{`"9223372036854775807"`, `9223372036854775807`} { + r := httptest.NewRequest("POST", "/", strings.NewReader(`{"id":`+id+`,"dynamic":{"n":9223372036854775807},"ids":["1",2]}`)) + r.Header.Set("Content-Type", "application/json") + var req request + if err = b.Bind(r, nil, &req); err != nil { + t.Fatal(err) + } + if req.ID != flexibleID(9223372036854775807) || req.Dynamic.(map[string]any)["n"] != json.Number("9223372036854775807") { + t.Fatalf("precision: %+v", req) + } + encoded, _ := json.Marshal(req) + if !strings.Contains(string(encoded), `"id":"9223372036854775807"`) { + t.Fatal(string(encoded)) + } + } + for _, body := range []string{`{"id":true,"dynamic":null}`, `{"id":1.1,"dynamic":null}`, `{"id":9223372036854775808,"dynamic":null}`, `{"id":1,"dynamic":{"x":1,"x":2}}`, `{"id":1,"dynamic":null,"unexpected":1}`} { + r := httptest.NewRequest("POST", "/", strings.NewReader(body)) + r.Header.Set("Content-Type", "application/json") + req := request{ID: 7} + if b.Bind(r, nil, &req) == nil || req.ID != 7 { + t.Fatalf("accepted or mutated %s", body) + } + } + for _, value := range []string{`null`, `[1,"x"]`, `false`, `"text"`} { + r := httptest.NewRequest("POST", "/", strings.NewReader(`{"id":1,"dynamic":`+value+`}`)) + r.Header.Set("Content-Type", "application/json") + if err := b.Bind(r, nil, &request{}); err != nil { + t.Fatal(err) + } + } + if _, err := New[request](); err == nil { + t.Fatal("implicit custom type registration") + } + response, err := b.ResponseSchema() + if err != nil { + t.Fatal(err) + } + if response.Properties["id"].Type != "string" || response.Properties["ids"].Items.Type != "string" || len(b.BodySchema().Properties["id"].AnyOf) != 2 { + t.Fatal("input/output schemas conflated") + } + type path struct { + ID flexibleID `path:"id"` + } + p, err := New[path](scalar) + if err != nil { + t.Fatal(err) + } + var req path + if err := p.Bind(httptest.NewRequest("GET", "/", nil), map[string]string{"id": "9223372036854775807"}, &req); err != nil { + t.Fatal(err) + } + if p.Parameters()[0].Schema.Type != "string" { + t.Fatal("text parameters need text schema") + } + if _, err := New[path](ScalarOf[flexibleID]("string|int64", "string", false)); err == nil { + t.Fatal("text support not explicit") + } + schemaOnly, err := Describe(reflect.TypeOf(request{}), scalar) + if err != nil { + t.Fatal(err) + } + if schemaOnly.Bind(httptest.NewRequest("POST", "/", nil), nil, &request{}) == nil { + t.Fatal("schema-only binder executed") + } +} + +func TestScalarPointersCollectionsAndRegistration(t *testing.T) { + scalar := ScalarOf[flexibleID]("string|int64", "string", true) + type nullable struct { + ID *flexibleID `json:"id,optional"` + Values map[string]*flexibleID `json:"values,optional"` + } + b, err := New[nullable](scalar) + if err != nil { + t.Fatal(err) + } + for _, body := range []string{`{}`, `{"id":null}`, `{"id":1,"values":{"a":"2","b":null}}`} { + r := httptest.NewRequest("POST", "/", strings.NewReader(body)) + r.Header.Set("Content-Type", "application/json") + if err := b.Bind(r, nil, &nullable{}); err != nil { + t.Fatal(err) + } + } + input := b.BodySchema().Properties["id"] + if len(input.AnyOf) != 3 || !input.AnyOf[2].Nullable || len(input.AnyOf[2].Enum) != 1 || input.AnyOf[2].Enum[0] != nil { + t.Fatal("nullable input union does not describe null") + } + out, err := b.ResponseSchema() + if err != nil { + t.Fatal(err) + } + if out.Properties["id"].Type != "string" || !out.Properties["id"].Nullable { + t.Fatal("nullable output type") + } + type query struct { + IDs []flexibleID `form:"ids"` + } + q, err := New[query](scalar) + if err != nil { + t.Fatal(err) + } + var req query + if err := q.Bind(httptest.NewRequest("GET", "/?ids=1&ids=2", nil), nil, &req); err != nil || !reflect.DeepEqual(req.IDs, []flexibleID{1, 2}) { + t.Fatalf("query scalar collection: %+v %v", req, err) + } + if q.Parameters()[0].Schema.Items.Type != "string" { + t.Fatal("text array schema") + } + if _, err := New[query](ScalarOf[flexibleID]("string|int64", "string", false)); err == nil { + t.Fatal("array bypassed text declaration") + } + if err := ValidateScalars(ScalarOf[int64]("int64", "string", true)); err == nil { + t.Fatal("missing methods accepted") + } + if err := ValidateScalars(scalar, scalar); err == nil { + t.Fatal("duplicate registration") + } + type strict struct { + ID int64 `json:"id"` + } + strictBinder := MustNew[strict]() + r := httptest.NewRequest("POST", "/", strings.NewReader(`{"id":"1"}`)) + r.Header.Set("Content-Type", "application/json") + if strictBinder.Bind(r, nil, &strict{}) == nil { + t.Fatal("other binder globally loosened") + } +} + +type panicScalar string + +func (v panicScalar) MarshalJSON() ([]byte, error) { return json.Marshal(string(v)) } +func (v *panicScalar) UnmarshalJSON([]byte) error { panic("decoder failure") } +func TestScalarPanicIsBindingError(t *testing.T) { + type req struct { + Value panicScalar `json:"value"` + } + b := MustNew[req](ScalarOf[panicScalar]("string", "string", false)) + r := httptest.NewRequest("POST", "/", strings.NewReader(`{"value":"bad"}`)) + r.Header.Set("Content-Type", "application/json") + dst := req{Value: "original"} + if err := b.Bind(r, nil, &dst); err == nil || dst.Value != "original" { + t.Fatalf("decoder panic: %+v %v", dst, err) + } +} + +func TestLimitedJSONTrailingWhitespace(t *testing.T) { + type req struct { + Value any `json:"value"` + } + b := MustNew[req]() + r := httptest.NewRequest("POST", "/", strings.NewReader(`{"value":null}`+strings.Repeat(" ", 65))) + r.Header.Set("Content-Type", "application/json") + r.ContentLength = -1 + r.Body = http.MaxBytesReader(httptest.NewRecorder(), r.Body, 64) + err := b.Bind(r, nil, &req{}, 64) + if apierrors.From(err).Status != 413 { + t.Fatalf("tail limit error: %v", err) + } +} + +func TestResponseSchemaPreservesOrdinaryPointerConstraints(t *testing.T) { + type dto struct { + Count *int `json:"count,optional,range=[1:10]"` + Choice *string `json:"choice,options=a|b"` + Values *[]int `json:"values,optional" binding:"max=3"` + } + b := MustNew[dto]() + response, err := b.ResponseSchema() + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(response, b.BodySchema()) { + t.Fatalf("changed ordinary response constraints: %#v", response) + } +} diff --git a/examples/compatibility/ids/id.go b/examples/compatibility/ids/id.go new file mode 100644 index 0000000..23c093e --- /dev/null +++ b/examples/compatibility/ids/id.go @@ -0,0 +1,29 @@ +// Package ids demonstrates an application-owned scalar. The framework does not +// impose this ID policy on any other type or project. +package ids + +import ( + "encoding/json" + "strconv" +) + +type ID int64 + +func (id ID) MarshalJSON() ([]byte, error) { return json.Marshal(strconv.FormatInt(int64(id), 10)) } +func (id *ID) UnmarshalJSON(raw []byte) error { + value := string(raw) + if len(raw) > 0 && raw[0] == '"' { + if err := json.Unmarshal(raw, &value); err != nil { + return err + } + } + return id.UnmarshalText([]byte(value)) +} +func (id *ID) UnmarshalText(raw []byte) error { + value, err := strconv.ParseInt(string(raw), 10, 64) + if err != nil { + return err + } + *id = ID(value) + return nil +} diff --git a/examples/compatibility/main.go b/examples/compatibility/main.go new file mode 100644 index 0000000..c168870 --- /dev/null +++ b/examples/compatibility/main.go @@ -0,0 +1,75 @@ +// This example only uses local HTTP and bounded streaming; it has no database. +package main + +import ( + "io" + "log" + "net/http" + "strings" + + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/contract" + "git.yuntuo.hk/k2safe/go-omni/examples/compatibility/ids" + "git.yuntuo.hk/k2safe/go-omni/httpx" +) + +type echoRequest struct { + ID ids.ID `json:"id"` + Data any `json:"data"` +} + +func main() { + app := omni.New(omni.WithLogger(true)) + defer app.Close() + app.Use(httpx.WrapMiddleware(func(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("X-Example", "compatibility") + next.ServeHTTP(w, r) + }) + })) + app.GET("/legacy", httpx.WrapHandler(http.HandlerFunc(http.NotFound))) + binder := contract.MustNew[echoRequest](contract.ScalarOf[ids.ID]("string|int64", "string", true)) + app.POST("/echo", func(c *httpx.Context) error { + var req echoRequest + if err := binder.Bind(c.Request(), nil, &req); err != nil { + return err + } + return c.OK(req) + }) + app.HandleWithBodyLimit("POST", "/upload", 10<<20, nil, func(c *httpx.Context) error { + reader, err := c.Request().MultipartReader() + if err != nil { + return omni.BadRequest("INVALID_MULTIPART", "multipart body required") + } + var total int64 + for { + part, err := reader.NextPart() + if err == io.EOF { + break + } + if err != nil { + return err + } + // Replace io.Discard with a streaming storage writer. Do not trust file names + // as filesystem paths. io.Copy never buffers the complete uploaded file. + n, copyErr := io.Copy(io.Discard, part) + closeErr := part.Close() + if copyErr != nil { + return copyErr + } + if closeErr != nil { + return closeErr + } + total += n + } + return c.OK(map[string]int64{"bytes": total}) + }) + app.GET("/download", httpx.WrapHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/octet-stream") + w.Header().Set("Content-Disposition", `attachment; filename="example.txt"`) + _, _ = io.Copy(w, strings.NewReader("A standard Handler keeps its original download response.\n")) + }))) + if err := app.Run("127.0.0.1:8080"); err != nil { + log.Fatal(err) + } +} diff --git a/examples/contracts/compatibility.api b/examples/contracts/compatibility.api new file mode 100644 index 0000000..46b0c16 --- /dev/null +++ b/examples/contracts/compatibility.api @@ -0,0 +1,64 @@ +syntax = "v1" + +scalar BusinessID ( + goImport: "git.yuntuo.hk/k2safe/go-omni/examples/compatibility/ids" + goType: ID + jsonInput: string | int64 + jsonOutput: string + text: true +) + +type EchoReq { + ID BusinessID `json:"id"` + Data any `json:"data"` + Legacy interface{} `json:"legacy,optional"` +} +type EchoResp { + ID BusinessID `json:"id"` + Data any `json:"data"` +} +type ItemReq { + ID BusinessID `path:"id"` +} +type CustomReq { + Value string `json:"value"` +} +type CustomResp { + Value string `json:"value"` +} + +@server (group: demo) +service compatibility-api { + @body (maxBytes: 2097152) + @handler echo + post /echo (EchoReq) returns (EchoResp) + + @handler item + get /items/:id (ItemReq) returns (EchoResp) + + @binding custom + @body (maxBytes: 512) + @handler custom + post /custom (CustomReq) returns (CustomResp) + + @handlerMode manual + @body (mode: multipart + maxBytes: 65536) + @handler upload + post /upload + + @handlerMode manual + @body (mode: raw + maxBytes: 128) + @produces "text/plain" + @handler callback + post /callback +} + +@server (group: files) +service compatibility-api { + @handlerMode manual + @produces "application/octet-stream" + @handler download + get /download +} diff --git a/examples/userService/.goOmni/apiManifest.json b/examples/userService/.goOmni/apiManifest.json index d17197a..2de15da 100644 --- a/examples/userService/.goOmni/apiManifest.json +++ b/examples/userService/.goOmni/apiManifest.json @@ -13,8 +13,8 @@ "internal/apierrors/errorsGen.go": "20e94bbc90ae9f4330de4c25b9a550026c4c352502719a643ede3e8912a6566e", "internal/handler/openapiGen.go": "e4130f5c7494f7db4830bfef203fa8154a9cd087c95954f64c29d74693ead433", "internal/handler/responseGen.go": "d742efc2a3d1d168fae7849e905297bd440c3d6b3802b6150f494c07e6097f4f", - "internal/handler/routesGen.go": "0ff429a8aa7f8643e0e844f7c43a63a1f82cca5ce6bfe917f06edd860ebe923f", - "internal/handler/user/handlersGen.go": "06f3c2798ff323812ff3eb2d8b933a569af3ce1b86a67b66036374add98c0d7e", + "internal/handler/routesGen.go": "4e0c6879cea40e9c0155439fa1fce2cd19b5f83ea6e304523c41f8dc9c3a28e0", + "internal/handler/user/handlersGen.go": "1808afc8e4e8ca9785a8552eff64fad39bd5fe4e6e92971f35afc9006e23d76e", "internal/middleware/registryGen.go": "a6122a5453dedc6a4415860108814c50e155128cfe5412b39dd33ce2d01b4b00", "internal/types/typesGen.go": "3ab2ce9ffe467c3ea53ac3d1fec283ecebc87cef57fd592d08dca991fd964876" } diff --git a/examples/userService/internal/handler/routesGen.go b/examples/userService/internal/handler/routesGen.go index 66c018d..6cfb318 100644 --- a/examples/userService/internal/handler/routesGen.go +++ b/examples/userService/internal/handler/routesGen.go @@ -17,7 +17,7 @@ func RegisterRoutes(router openapi.Registrar, deps *servicecontext.Context) erro if deps.Middlewares["Auth"] == nil { return fmt.Errorf("missing middleware Auth") } - h0 := h0.New(s0.New(deps)) + h0 := h0.New(s0.New(deps), deps) httpx.HandleWithResponse(router, "POST", "/api/v1/users", responseEncoder("", false), h0.CreateUser, deps.Middlewares["Auth"]) httpx.HandleWithResponse(router, "GET", "/api/v1/users/:id", responseEncoder("", false), h0.GetUser, deps.Middlewares["Auth"]) httpx.HandleWithResponse(router, "GET", "/api/v1/users", responseEncoder("", false), h0.ListUsers, deps.Middlewares["Auth"]) diff --git a/examples/userService/internal/handler/user/handlersGen.go b/examples/userService/internal/handler/user/handlersGen.go index 3ad35dd..3cc5e1b 100644 --- a/examples/userService/internal/handler/user/handlersGen.go +++ b/examples/userService/internal/handler/user/handlersGen.go @@ -3,6 +3,7 @@ package user import ( "context" + "example.com/userService/internal/servicecontext" "example.com/userService/internal/types" "fmt" "git.yuntuo.hk/k2safe/go-omni/contract" @@ -17,9 +18,18 @@ type Service interface { DeleteUser(ctx context.Context, req *types.UserIDReq) error ExportUser(ctx context.Context, req *types.UserIDReq) (*types.ExportResp, error) } -type Handler struct{ service Service } +type Handler struct { + service Service + deps *servicecontext.Context +} -func New(s Service) *Handler { return &Handler{service: s} } +func New(s Service, deps ...*servicecontext.Context) *Handler { + h := &Handler{service: s} + if len(deps) > 0 { + h.deps = deps[0] + } + return h +} var createUserRequest = contract.MustNew[types.CreateUserReq]() diff --git a/httpx/body.go b/httpx/body.go new file mode 100644 index 0000000..5bce17c --- /dev/null +++ b/httpx/body.go @@ -0,0 +1,60 @@ +package httpx + +import ( + "errors" + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" + "git.yuntuo.hk/k2safe/go-omni/response" + "net/http" +) + +// LimitBody is useful for handwritten routes. Built-in App.HandleWithBodyLimit +// additionally installs the reader cap before global middleware can read it. +func LimitBody(maxBytes int64) Middleware { + if maxBytes <= 0 || maxBytes > 1<<40 { + panic("httpx: body limit must be 1..1 TiB") + } + return func(next Handler) Handler { + return func(c *Context) error { + r := c.Request() + if r.ContentLength > maxBytes { + return bodyTooLarge() + } + if r.Body != nil { + r.Body = http.MaxBytesReader(c.Raw(), r.Body, maxBytes) + } + defer CleanupMultipart(r) + err := next(c) + var limitErr *http.MaxBytesError + if errors.As(err, &limitErr) { + return bodyTooLarge() + } + return err + } + } +} +func bodyTooLarge() error { return apierrors.New(413, "BODY_TOO_LARGE", "request body exceeds limit") } + +// CleanupMultipart removes disk spill files created by ParseMultipartForm. +// Callers still close individual multipart files. Streaming MultipartReader +// handlers own any files they create themselves. +func CleanupMultipart(r *http.Request) { + if r != nil && r.MultipartForm != nil { + _ = r.MultipartForm.RemoveAll() + } +} + +type BodyLimitRegistrar interface { + HandleWithBodyLimit(string, string, int64, response.Encoder, Handler, ...Middleware) +} + +func HandleWithBodyLimit(router interface { + Handle(string, string, Handler, ...Middleware) +}, method, path string, maxBytes int64, encoder response.Encoder, handler Handler, middleware ...Middleware) { + if r, ok := router.(BodyLimitRegistrar); ok { + r.HandleWithBodyLimit(method, path, maxBytes, encoder, handler, middleware...) + return + } + // Custom registrars without the early hook cap at route middleware entry. + all := append([]Middleware{LimitBody(maxBytes)}, middleware...) + HandleWithResponse(router, method, path, encoder, handler, all...) +} diff --git a/httpx/bridge.go b/httpx/bridge.go new file mode 100644 index 0000000..89a0e80 --- /dev/null +++ b/httpx/bridge.go @@ -0,0 +1,80 @@ +package httpx + +import ( + "context" + "net/http" +) + +type bridgeState struct{ parent *Context } + +// WrapHandler runs an existing standard handler inside the App middleware chain. +// It preserves the original response, including downloads and streaming output. +func WrapHandler(handler http.Handler) Handler { + if handler == nil { + panic("httpx: nil HTTP handler") + } + return func(c *Context) error { + r := bridgeRequest(c) + defer CleanupMultipart(r) + handler.ServeHTTP(c.Raw(), r) + return nil + } +} + +// WrapMiddleware adapts synchronous standard HTTP middleware. The middleware is +// constructed once per chain, preserving its shared state across requests. +// Errors from next are encoded before unwinding so response-transforming +// middleware sees them. Preserve the incoming request context (derive with +// WithContext), and never call next after ServeHTTP returns, as with net/http. +func WrapMiddleware(middleware func(http.Handler) http.Handler) Middleware { + if middleware == nil { + panic("httpx: nil HTTP middleware") + } + return func(next Handler) Handler { + // Each nested bridge has its own key, including repeated use of one adapter. + key := &bridgeState{} + wrapped := middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + state, ok := r.Context().Value(key).(*bridgeState) + if !ok { + panic("httpx: standard middleware must preserve the incoming request context") + } + c := state.parent + child := c.Clone(r.Context(), w) + child.request = r + if c.Responded() { + child.writer.(interface{ inheritCommit(int) }).inheritCommit(c.Status()) + } + defer CleanupMultipart(r) + defer func() { c.values = child.values; c.encoder = child.encoder }() + if err := next(child); err != nil && !c.Responded() { + child.WriteError(err) + } + })) + if wrapped == nil { + panic("httpx: HTTP middleware returned nil handler") + } + return func(c *Context) error { + r := bridgeRequest(c).WithContext(context.WithValue(c.Context(), key, &bridgeState{parent: c})) + defer CleanupMultipart(r) + wrapped.ServeHTTP(c.Raw(), r) + return nil + } + } +} + +// WrapFuncMiddleware supports middleware shaped as func(http.HandlerFunc) http.HandlerFunc. +func WrapFuncMiddleware(middleware func(http.HandlerFunc) http.HandlerFunc) Middleware { + if middleware == nil { + panic("httpx: nil HTTP middleware") + } + return WrapMiddleware(func(next http.Handler) http.Handler { return middleware(next.ServeHTTP) }) +} + +// Populate Go 1.22 path values without mutating a parent router's request maps. +func bridgeRequest(c *Context) *http.Request { + r := c.Request().Clone(c.Context()) + for name, value := range c.params { + r.SetPathValue(name, value) + } + return r +} diff --git a/httpx/bridge_test.go b/httpx/bridge_test.go new file mode 100644 index 0000000..ecfd2e1 --- /dev/null +++ b/httpx/bridge_test.go @@ -0,0 +1,363 @@ +package httpx_test + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "mime/multipart" + "net" + "net/http" + "net/http/httptest" + "os" + "strings" + "testing" + "time" + + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/logx" +) + +func TestLegacyResponseTrackingAndMiddleware(t *testing.T) { + var logs bytes.Buffer + closeLog, err := logx.Setup(logx.Config{Writer: &logs, Format: "json"}) + if err != nil { + t.Fatal(err) + } + defer closeLog() + app := omni.New(omni.WithLogger(true)) + app.GET("/legacy", httpx.WrapHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { http.NotFound(w, r) }))) + app.GET("/late", func(c *httpx.Context) error { + c.Raw().WriteHeader(404) + _, _ = c.Raw().Write([]byte("original")) + return errors.New("late error") + }) + app.GET("/panic", func(c *httpx.Context) error { _, _ = c.Raw().Write([]byte("partial")); panic("late panic") }) + for _, tc := range []struct { + path string + status int + body string + }{{"/legacy", 404, "404 page not found\n"}, {"/late", 404, "original"}, {"/panic", 200, "partial"}} { + w := httptest.NewRecorder() + app.ServeHTTP(w, httptest.NewRequest("GET", tc.path, nil)) + if w.Code != tc.status || w.Body.String() != tc.body { + t.Fatalf("%s: %d %q", tc.path, w.Code, w.Body.String()) + } + } + count := 0 + for _, line := range strings.Split(logs.String(), "\n") { + var e map[string]any + if json.Unmarshal([]byte(line), &e) != nil || e["msg"] != "http.request" { + continue + } + count++ + if e["path"] != "/panic" && e["status"] != float64(404) { + t.Fatalf("wrong log: %s", line) + } + } + if count != 3 { + t.Fatalf("access logs: %d", count) + } + + type key struct{} + app = omni.New() + app.Use(httpx.WrapFuncMiddleware(func(next http.HandlerFunc) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("X-Before", "yes") + next(w, r.WithContext(context.WithValue(r.Context(), key{}, "identity"))) + } + })) + app.GET("/context", func(c *httpx.Context) error { + if c.Context().Value(key{}) != "identity" || c.RequestID() == "" { + t.Error("lost context") + } + return omni.NotFound("MISSING", "missing") + }) + w := httptest.NewRecorder() + app.ServeHTTP(w, httptest.NewRequest("GET", "/context", nil)) + if w.Code != 404 || w.Header().Get("X-Before") != "yes" || strings.Count(w.Body.String(), "MISSING") != 1 { + t.Fatalf("middleware response: %d %s", w.Code, w.Body) + } +} + +type plainWriter struct { + header http.Header + status int +} + +func (w *plainWriter) Header() http.Header { return w.header } +func (w *plainWriter) WriteHeader(status int) { w.status = status } +func (w *plainWriter) Write(p []byte) (int, error) { return len(p), nil } +func TestWriterCapabilitiesAndInformationalStatus(t *testing.T) { + w := &plainWriter{header: http.Header{}} + c := httpx.NewContext(httptest.NewRequest("GET", "/", nil), w, nil) + if _, ok := c.Raw().(http.Flusher); ok { + t.Fatal("invented Flusher") + } + if _, ok := c.Raw().(http.Hijacker); ok { + t.Fatal("invented Hijacker") + } + if err := http.NewResponseController(c.Raw()).Flush(); !errors.Is(err, http.ErrNotSupported) { + t.Fatalf("Flush: %v", err) + } + c.Raw().WriteHeader(103) + if c.Responded() { + t.Fatal("103 committed final response") + } + c.Raw().WriteHeader(201) + c.Raw().WriteHeader(500) + if !c.Responded() || c.Status() != 201 || w.status != 201 { + t.Fatal("first final status not preserved") + } + recorder := httptest.NewRecorder() + c = httpx.NewContext(httptest.NewRequest("GET", "/", nil), recorder, nil) + if err := http.NewResponseController(c.Raw()).Flush(); err != nil { + t.Fatal(err) + } + if !c.Responded() || c.Status() != 200 || !recorder.Flushed { + t.Fatal("flush not tracked") + } +} + +func TestLegacyStreamingAcrossTransports(t *testing.T) { + for _, transport := range []string{omni.TransportNetHTTP, omni.TransportNative} { + t.Run(transport, func(t *testing.T) { + release := make(chan struct{}) + defer close(release) + app := omni.New(omni.WithTransport(transport)) + app.GET("/stream", httpx.WrapHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/plain") + _, _ = io.WriteString(w, "first\n") + if err := http.NewResponseController(w).Flush(); err != nil { + t.Error(err) + return + } + select { + case <-release: + case <-r.Context().Done(): + } + }))) + app.GET("/file", httpx.WrapHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Disposition", `attachment; filename="data.bin"`) + _, _ = io.Copy(w, strings.NewReader(strings.Repeat("x", 2<<20))) + }))) + app.HandleWithBodyLimit("POST", "/bounded", 32, nil, func(c *httpx.Context) error { + _, err := io.Copy(io.Discard, c.Request().Body) + if err != nil { + return err + } + return c.NoContent() + }) + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { done <- app.ServeContext(ctx, listener) }() + defer func() { + cancel() + select { + case <-done: + case <-time.After(15 * time.Second): + t.Error("shutdown timed out") + } + }() + client := &http.Client{Timeout: 3 * time.Second} + defer client.CloseIdleConnections() + for _, body := range []io.Reader{strings.NewReader(strings.Repeat("x", 33)), io.NopCloser(strings.NewReader(strings.Repeat("x", 33)))} { + res, err := client.Post("http://"+listener.Addr().String()+"/bounded", "application/octet-stream", body) + if err != nil { + t.Fatal(err) + } + _, _ = io.Copy(io.Discard, res.Body) + res.Body.Close() + if res.StatusCode != 413 { + t.Fatalf("real transport body limit: %d", res.StatusCode) + } + } + res, err := client.Get("http://" + listener.Addr().String() + "/stream") + if err != nil { + t.Fatal(err) + } + first := make([]byte, 6) + _, err = io.ReadFull(res.Body, first) + res.Body.Close() + if err != nil || string(first) != "first\n" { + t.Fatalf("stream was buffered: %s %v", first, err) + } + res, err = client.Get("http://" + listener.Addr().String() + "/file") + if err != nil { + t.Fatal(err) + } + size, err := io.Copy(io.Discard, res.Body) + res.Body.Close() + if err != nil || size != 2<<20 || res.Header.Get("Content-Disposition") == "" { + t.Fatalf("file: %d %v", size, err) + } + }) + } +} + +func TestStandardMiddlewareStateAndShortCircuit(t *testing.T) { + constructions, calls := 0, 0 + app := omni.New() + defer app.Close() + app.Use(httpx.WrapMiddleware(func(next http.Handler) http.Handler { + constructions++ + count := 0 + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + count++ + if count == 2 { + http.Error(w, "denied", 401) + return + } + next.ServeHTTP(w, r) + }) + })) + app.GET("/", func(c *httpx.Context) error { calls++; return c.NoContent() }) + for _, status := range []int{204, 401, 204} { + w := httptest.NewRecorder() + app.ServeHTTP(w, httptest.NewRequest("GET", "/", nil)) + if w.Code != status { + t.Fatalf("got %d want %d", w.Code, status) + } + } + if constructions != 1 || calls != 2 { + t.Fatalf("middleware lifecycle: %d constructions, %d calls", constructions, calls) + } +} + +func TestStandardMiddlewareTransformsErrorAndPreservesCancellation(t *testing.T) { + app := omni.New() + defer app.Close() + app.Use(httpx.WrapMiddleware(func(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + // A legacy middleware may buffer its own small response. The bridge must + // report its eventual physical status, and encode errors before it unwinds. + temp := httptest.NewRecorder() + next.ServeHTTP(temp, r) + if temp.Code != 404 { + t.Errorf("inner error not encoded: %d", temp.Code) + } + w.WriteHeader(409) + _, _ = io.WriteString(w, strings.ToUpper(temp.Body.String())) + }) + })) + app.GET("/", func(c *httpx.Context) error { + if !errors.Is(c.Context().Err(), context.Canceled) { + t.Error("cancellation lost") + } + return omni.NotFound("missing", "missing") + }) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + w := httptest.NewRecorder() + app.ServeHTTP(w, httptest.NewRequest("GET", "/", nil).WithContext(ctx)) + if w.Code != 409 || !strings.Contains(w.Body.String(), "MISSING") { + t.Fatalf("%d %s", w.Code, w.Body) + } +} + +func TestRouteBodyLimitBeforeMiddleware(t *testing.T) { + for _, length := range []int64{-1, 33} { + t.Run(fmt.Sprint(length), func(t *testing.T) { + app := omni.New() + defer app.Close() + read := 0 + app.Use(func(next httpx.Handler) httpx.Handler { + return func(c *httpx.Context) error { + body, err := io.ReadAll(c.Request().Body) + read = len(body) + if err != nil { + return err + } + return next(c) + } + }) + app.Group("/api").HandleWithBodyLimit("POST", "/upload", 32, nil, func(c *httpx.Context) error { t.Error("called oversized handler"); return nil }) + r := httptest.NewRequest("POST", "/api/upload", strings.NewReader(strings.Repeat("x", 33))) + r.ContentLength = length + w := httptest.NewRecorder() + app.ServeHTTP(w, r) + if w.Code != 413 || read > 32 { + t.Fatalf("limit: status=%d read=%d", w.Code, read) + } + }) + } +} + +func TestMultipartCleanupOnPanic(t *testing.T) { + temp := t.TempDir() + t.Setenv("TMPDIR", temp) + var body bytes.Buffer + form := multipart.NewWriter(&body) + part, err := form.CreateFormFile("file", "x.bin") + if err != nil { + t.Fatal(err) + } + _, _ = part.Write(bytes.Repeat([]byte("x"), 4096)) + _ = form.Close() + app := omni.New() + defer app.Close() + app.HandleWithBodyLimit("POST", "/", 8192, nil, httpx.WrapHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if err := r.ParseMultipartForm(1); err != nil { + t.Error(err) + return + } + panic("after disk spill") + }))) + r := httptest.NewRequest("POST", "/", &body) + r.Header.Set("Content-Type", form.FormDataContentType()) + w := httptest.NewRecorder() + app.ServeHTTP(w, r) + entries, err := os.ReadDir(temp) + if w.Code != 500 || err != nil || len(entries) != 0 { + t.Fatalf("cleanup: %d %v %v", w.Code, entries, err) + } +} + +func TestStandardMiddlewareCommitsBeforeCallingNext(t *testing.T) { + app := omni.New() + defer app.Close() + app.Use(httpx.WrapMiddleware(func(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(202) + _, _ = io.WriteString(w, "accepted") + next.ServeHTTP(w, r) + }) + })) + app.GET("/", func(c *httpx.Context) error { + if !c.Responded() || c.Status() != 202 { + t.Error("lost middleware commit") + } + return errors.New("late failure") + }) + w := httptest.NewRecorder() + app.ServeHTTP(w, httptest.NewRequest("GET", "/", nil)) + if w.Code != 202 || w.Body.String() != "accepted" { + t.Fatalf("late append: %d %q", w.Code, w.Body.String()) + } +} + +func TestBridgeStandardPathValues(t *testing.T) { + app := omni.New() + defer app.Close() + app.Use(httpx.WrapMiddleware(func(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.PathValue("itemId") != "abc" { + t.Error("middleware path value lost") + } + next.ServeHTTP(w, r) + }) + })) + app.GET("/items/:itemId", httpx.WrapHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = io.WriteString(w, r.PathValue("itemId")) }))) + w := httptest.NewRecorder() + app.ServeHTTP(w, httptest.NewRequest("GET", "/items/abc", nil)) + if w.Body.String() != "abc" { + t.Fatal("handler path value lost") + } +} diff --git a/httpx/context.go b/httpx/context.go index c6f66a1..adf884b 100644 --- a/httpx/context.go +++ b/httpx/context.go @@ -3,6 +3,7 @@ package httpx import ( "context" "encoding/json" + "errors" "net/http" apierrors "git.yuntuo.hk/k2safe/go-omni/errors" @@ -101,12 +102,15 @@ func NewContext(request *http.Request, writer http.ResponseWriter, params map[st if request == nil { request = &http.Request{} } - return &Context{ + c := &Context{ request: request, - writer: writer, params: params, values: make(map[string]any), } + if writer != nil { + c.writer = trackWriter(writer, c) + } + return c } func (c *Context) Request() *http.Request { return c.request } @@ -120,8 +124,8 @@ func (c *Context) Context() context.Context { return c.request.Context() } -// Raw returns the response writer supplied by the active network adapter. -// Prefer the transport-neutral methods on Context for portable handlers. +// Raw returns a streaming response writer that tracks commits and status. +// Optional HTTP capabilities are preserved only when the underlying writer supports them. func (c *Context) Raw() http.ResponseWriter { return c.writer } // Clone gives a buffering middleware its own response state and values map. @@ -146,7 +150,6 @@ func (c *Context) WriteResponse(status int, headers http.Header, body []byte) er for key, values := range headers { c.writer.Header()[key] = append([]string(nil), values...) } - c.responded, c.status = true, status c.writer.WriteHeader(status) if len(body) > 0 { if _, err := c.writer.Write(body); err != nil { @@ -216,8 +219,6 @@ func (c *Context) JSON(status int, value any) error { if err != nil { return apierrors.Internal(err) } - c.responded = true - c.status = status c.writer.Header().Set("Content-Type", "application/json; charset=utf-8") c.writer.WriteHeader(status) if _, err := c.writer.Write(append(data, '\n')); err != nil { @@ -243,8 +244,6 @@ func (c *Context) NoContent() error { if c.writer == nil { return apierrors.Internal(nil) } - c.responded = true - c.status = http.StatusNoContent c.writer.WriteHeader(http.StatusNoContent) return nil } @@ -271,12 +270,15 @@ func (c *Context) Status() int { func (c *Context) Responded() bool { return c.responded } // WriteError writes a stable public error response. Internal causes are not -// serialized. Responses committed through Context methods are preserved; -// direct Raw writes are outside this response-tracking contract. +// serialized. Responses committed through Context methods or Raw are preserved. func (c *Context) WriteError(err error) { if c.responded || err == nil { return } + var bodyErr *http.MaxBytesError + if errors.As(err, &bodyErr) { + err = bodyTooLarge() + } apiErr := apierrors.From(err) if writeErr := c.writeResult(response.Result{ Status: apiErr.Status, diff --git a/httpx/trackingWriter.go b/httpx/trackingWriter.go new file mode 100644 index 0000000..997850c --- /dev/null +++ b/httpx/trackingWriter.go @@ -0,0 +1,315 @@ +package httpx + +import ( + "bufio" + "io" + "net" + "net/http" +) + +// trackingWriter does not buffer response bodies. Each wrapper has its own +// commit flag, so a standard middleware may safely wrap an already tracked writer. +type trackingWriter struct { + http.ResponseWriter + context *Context + committed bool +} + +func (w *trackingWriter) Unwrap() http.ResponseWriter { return w.ResponseWriter } +func (w *trackingWriter) inheritCommit(status int) { w.mark(status) } +func (w *trackingWriter) mark(status int) { + w.committed = true + w.context.responded = true + w.context.status = status +} +func (w *trackingWriter) WriteHeader(status int) { + if w.committed { + return + } + // Let the server validate the status before updating the observation. + w.ResponseWriter.WriteHeader(status) + if status == 101 || status >= 200 { + w.mark(status) + } +} +func (w *trackingWriter) Write(p []byte) (int, error) { + if !w.committed { + w.mark(http.StatusOK) + } + return w.ResponseWriter.Write(p) +} + +type flushWriter struct{ *trackingWriter } + +func (w flushWriter) Flush() { _ = w.FlushError() } +func (w flushWriter) FlushError() error { + if !w.committed { + w.WriteHeader(http.StatusOK) + } + return http.NewResponseController(w.ResponseWriter).Flush() +} + +type hijackWriter struct{ *trackingWriter } + +func (w hijackWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) { + conn, rw, err := w.ResponseWriter.(http.Hijacker).Hijack() + if err == nil { + w.context.responded = true + w.committed = true + } + return conn, rw, err +} + +type pushWriter struct{ *trackingWriter } + +func (w pushWriter) Push(target string, options *http.PushOptions) error { + return w.ResponseWriter.(http.Pusher).Push(target, options) +} + +type readFromWriter struct{ *trackingWriter } + +func (w readFromWriter) ReadFrom(r io.Reader) (int64, error) { + // A writer-only view makes every write observable, including partial failures, + // while io.Copy retains bounded streaming memory and source WriterTo support. + return io.Copy(struct{ io.Writer }{w.trackingWriter}, r) +} + +type closeNotifyWriter struct{ *trackingWriter } + +func (w closeNotifyWriter) CloseNotify() <-chan bool { + return w.ResponseWriter.(http.CloseNotifier).CloseNotify() +} + +func trackWriter(writer http.ResponseWriter, c *Context) http.ResponseWriter { + w := &trackingWriter{ResponseWriter: writer, context: c} + mask := 0 + if _, ok := writer.(http.Flusher); ok { + mask |= 1 + } else if _, ok := writer.(interface{ FlushError() error }); ok { + mask |= 1 + } + if _, ok := writer.(http.Hijacker); ok { + mask |= 2 + } + if _, ok := writer.(http.Pusher); ok { + mask |= 4 + } + if _, ok := writer.(io.ReaderFrom); ok { + mask |= 8 + } + if _, ok := writer.(http.CloseNotifier); ok { + mask |= 16 + } + switch mask { + case 1: + return struct { + *trackingWriter + flushCapability + }{w, flushWriter{w}} + case 2: + return struct { + *trackingWriter + http.Hijacker + }{w, hijackWriter{w}} + case 3: + return struct { + *trackingWriter + flushCapability + http.Hijacker + }{w, flushWriter{w}, hijackWriter{w}} + case 4: + return struct { + *trackingWriter + http.Pusher + }{w, pushWriter{w}} + case 5: + return struct { + *trackingWriter + flushCapability + http.Pusher + }{w, flushWriter{w}, pushWriter{w}} + case 6: + return struct { + *trackingWriter + http.Hijacker + http.Pusher + }{w, hijackWriter{w}, pushWriter{w}} + case 7: + return struct { + *trackingWriter + flushCapability + http.Hijacker + http.Pusher + }{w, flushWriter{w}, hijackWriter{w}, pushWriter{w}} + case 8: + return struct { + *trackingWriter + io.ReaderFrom + }{w, readFromWriter{w}} + case 9: + return struct { + *trackingWriter + flushCapability + io.ReaderFrom + }{w, flushWriter{w}, readFromWriter{w}} + case 10: + return struct { + *trackingWriter + http.Hijacker + io.ReaderFrom + }{w, hijackWriter{w}, readFromWriter{w}} + case 11: + return struct { + *trackingWriter + flushCapability + http.Hijacker + io.ReaderFrom + }{w, flushWriter{w}, hijackWriter{w}, readFromWriter{w}} + case 12: + return struct { + *trackingWriter + http.Pusher + io.ReaderFrom + }{w, pushWriter{w}, readFromWriter{w}} + case 13: + return struct { + *trackingWriter + flushCapability + http.Pusher + io.ReaderFrom + }{w, flushWriter{w}, pushWriter{w}, readFromWriter{w}} + case 14: + return struct { + *trackingWriter + http.Hijacker + http.Pusher + io.ReaderFrom + }{w, hijackWriter{w}, pushWriter{w}, readFromWriter{w}} + case 15: + return struct { + *trackingWriter + flushCapability + http.Hijacker + http.Pusher + io.ReaderFrom + }{w, flushWriter{w}, hijackWriter{w}, pushWriter{w}, readFromWriter{w}} + case 16: + return struct { + *trackingWriter + http.CloseNotifier + }{w, closeNotifyWriter{w}} + case 17: + return struct { + *trackingWriter + flushCapability + http.CloseNotifier + }{w, flushWriter{w}, closeNotifyWriter{w}} + case 18: + return struct { + *trackingWriter + http.Hijacker + http.CloseNotifier + }{w, hijackWriter{w}, closeNotifyWriter{w}} + case 19: + return struct { + *trackingWriter + flushCapability + http.Hijacker + http.CloseNotifier + }{w, flushWriter{w}, hijackWriter{w}, closeNotifyWriter{w}} + case 20: + return struct { + *trackingWriter + http.Pusher + http.CloseNotifier + }{w, pushWriter{w}, closeNotifyWriter{w}} + case 21: + return struct { + *trackingWriter + flushCapability + http.Pusher + http.CloseNotifier + }{w, flushWriter{w}, pushWriter{w}, closeNotifyWriter{w}} + case 22: + return struct { + *trackingWriter + http.Hijacker + http.Pusher + http.CloseNotifier + }{w, hijackWriter{w}, pushWriter{w}, closeNotifyWriter{w}} + case 23: + return struct { + *trackingWriter + flushCapability + http.Hijacker + http.Pusher + http.CloseNotifier + }{w, flushWriter{w}, hijackWriter{w}, pushWriter{w}, closeNotifyWriter{w}} + case 24: + return struct { + *trackingWriter + io.ReaderFrom + http.CloseNotifier + }{w, readFromWriter{w}, closeNotifyWriter{w}} + case 25: + return struct { + *trackingWriter + flushCapability + io.ReaderFrom + http.CloseNotifier + }{w, flushWriter{w}, readFromWriter{w}, closeNotifyWriter{w}} + case 26: + return struct { + *trackingWriter + http.Hijacker + io.ReaderFrom + http.CloseNotifier + }{w, hijackWriter{w}, readFromWriter{w}, closeNotifyWriter{w}} + case 27: + return struct { + *trackingWriter + flushCapability + http.Hijacker + io.ReaderFrom + http.CloseNotifier + }{w, flushWriter{w}, hijackWriter{w}, readFromWriter{w}, closeNotifyWriter{w}} + case 28: + return struct { + *trackingWriter + http.Pusher + io.ReaderFrom + http.CloseNotifier + }{w, pushWriter{w}, readFromWriter{w}, closeNotifyWriter{w}} + case 29: + return struct { + *trackingWriter + flushCapability + http.Pusher + io.ReaderFrom + http.CloseNotifier + }{w, flushWriter{w}, pushWriter{w}, readFromWriter{w}, closeNotifyWriter{w}} + case 30: + return struct { + *trackingWriter + http.Hijacker + http.Pusher + io.ReaderFrom + http.CloseNotifier + }{w, hijackWriter{w}, pushWriter{w}, readFromWriter{w}, closeNotifyWriter{w}} + case 31: + return struct { + *trackingWriter + flushCapability + http.Hijacker + http.Pusher + io.ReaderFrom + http.CloseNotifier + }{w, flushWriter{w}, hijackWriter{w}, pushWriter{w}, readFromWriter{w}, closeNotifyWriter{w}} + } + return w +} + +type flushCapability interface { + Flush() + FlushError() error +} diff --git a/performance/compatibilityAcceptance.md b/performance/compatibilityAcceptance.md new file mode 100644 index 0000000..9026d14 --- /dev/null +++ b/performance/compatibilityAcceptance.md @@ -0,0 +1,24 @@ +# v0.9.0-preview 迁移扩展验收 + +本轮只使用本机临时服务与生成项目；没有运行业务服务器压测，没有修改 ePay/eTax 业务代码。 + +## 本地验收 + +- `GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=1 -count=1 -timeout=300s ./...`：40 个有测试的包通过；包含真实生成消费者、标准 Handler/Middleware、动态 JSON、路由限额和 scalar。 +- `go vet ./...`、`go build ./...`、gofmt、`git diff --check` 通过。 +- 独立 `examples/userService` module 的 race/vet/build 通过；生成文件 `--check` 通过，手写业务保持原样。 +- nethttp/native 本机真实 TCP 测试：首块数据立即 Flush、2 MiB 文件保持原响应、已知长度和 chunked 超限返回 413，并能继续处理正常请求。 +- 404 访问日志、提交后错误/panic 不追加 JSON、标准中间件提前写入/跨请求状态/取消/PathValue 通过。 +- multipart 落盘临时文件在成功和 panic 后清理；原始请求体 HMAC 验签、定制绑定和生成代码保护通过。 +- scalar 最大 int64、数字/字符串兼容、溢出/错误类型拒绝、指针 null、query 集合、输入/输出 schema 分离通过；普通字段仍严格。 +- ePay 原始契约只读校验：26 types、43 routes、0 errors、0 auth schemes。后两项是声明数量，不代表无需鉴权或没有业务错误；不代表 144 条注册路由已全部迁移。 + +环境：`go version go1.27.1 darwin/arm64`。记录时间：2026-09-23T17:47:26+00:00。 + +## 远程验收 + +发布后执行 `scripts/verifyCompatibilityRelease.py --version v0.9.0-preview`，结果生成到 `performance/compatibilityRemote.json`，校验从全新 module cache 安装的 CLI 与消费者固定依赖、无本地 replace、重生成保护和运行时行为。该结果在 main 单独记录，发布标签保持不可变。 + +## 范围 + +本次不是完整 ePay 迁移验收；资金事务、原权限体系、业务响应细节仍需业务自己的回归。未重跑 Linux soak、Nginx/Cloudflare 代理或云存储账号实网验收。 diff --git a/scripts/verifyCompatibilityRelease.py b/scripts/verifyCompatibilityRelease.py new file mode 100644 index 0000000..03fe268 --- /dev/null +++ b/scripts/verifyCompatibilityRelease.py @@ -0,0 +1,80 @@ +#!/usr/bin/env python3 +"""Install the released CLI in a fresh cache, then test its generated consumer. + +The fixture tests custom scalars, dynamic JSON, uploads, raw callback signatures, +downloads, body limits and regeneration protection. Only local temporary resources +are used; the framework and CLI are fetched from the specified remote version. +""" +import argparse +import hashlib +import json +import os +from pathlib import Path +import re +import subprocess +import tempfile +import time + +ROOT = Path(__file__).resolve().parents[1] +MODULE = "git.yuntuo.hk/k2safe/go-omni" + + +def run(args, cwd, env, timeout=300): + result = subprocess.run([str(arg) for arg in args], cwd=cwd, env=env, + capture_output=True, text=True, timeout=timeout) + output = re.sub(r"https://[^/@\s]+:[^/@\s]+@", "https://", result.stdout + result.stderr) + if result.returncode: + raise RuntimeError(f"{args[0]} failed ({result.returncode}):\n{output}") + return output + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--version", required=True) + parser.add_argument("--report", type=Path, default=ROOT / "performance/compatibilityRemote.json") + args = parser.parse_args() + if not re.fullmatch(r"v\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?", args.version): + parser.error("a fixed release version is required") + env = {k: v for k, v in os.environ.items() if not k.startswith("GO_OMNI_")} + env.update(GOWORK="off", GOFLAGS="", GOPRIVATE="git.yuntuo.hk", GONOSUMDB="git.yuntuo.hk") + start = time.monotonic() + with tempfile.TemporaryDirectory(prefix="goOmniCompatibilityRelease-") as directory: + temp = Path(directory) + (temp / "bin").mkdir() + env.update(GOBIN=str(temp / "bin"), GOMODCACHE=str(temp / "modcache")) + print(f"Installing CLI {args.version} in an empty module cache", flush=True) + run(["go", "install", MODULE + "/cmd/go-omni@" + args.version], temp, env) + cli = temp / "bin/go-omni" + run([cli, "api", "validate", "--api", ROOT / "examples/contracts/compatibility.api"], temp, env) + env.update(GO_OMNI_SCAFFOLD_INTEGRATION="1", GO_OMNI_RELEASE_CLI=str(cli), GO_OMNI_RELEASE_VERSION=args.version) + print("Testing remote generation, scalar binding, uploads and callback signatures", flush=True) + output = run(["go", "test", "-race", "-count=1", "-json", "-timeout=180s", + "./cmd/go-omni/apigen", "-run", "^TestCompatibilityConsumer$"], ROOT, env) + events = [json.loads(line) for line in output.splitlines() if line.startswith("{")] + passed = [e["Test"] for e in events if e.get("Action") == "pass" and "Test" in e] + assert passed == ["TestCompatibilityConsumer"], output + evidence = "".join(e.get("Output", "") for e in events) + checksum = re.search(r"checksum (h1:\S+); no local replace", evidence) + assert checksum, "consumer did not verify its remote dependency" + report = { + "passed": True, "frameworkVersion": args.version, "frameworkSum": checksum.group(1), + "freshModuleCache": True, "noLocalReplace": True, + "durationSeconds": round(time.monotonic() - start, 2), + "checks": ["remoteCLIInstall", "scalarContractValidation", "manualProjectGeneration", + "int64MaxValueOnWire", "dynamicJSON", "rawBodySignature", "routeBodyLimits", + "multipartTempCleanup", "unwrappedDownloads", + "handwrittenFilesPreserved", "generatedConsumerRaceTests", "fixedRemoteModuleChecksum"], + "sourceHashes": {name: hashlib.sha256((ROOT / name).read_bytes()).hexdigest() for name in + ("scripts/verifyCompatibilityRelease.py", "cmd/go-omni/apigen/compatibility_test.go", + "examples/contracts/compatibility.api", + "cmd/go-omni/apigen/testdata/compatibility/consumer_test.go", + "cmd/go-omni/apigen/testdata/compatibility/hooks.go", + "cmd/go-omni/apigen/testdata/compatibility/echo.go", + "cmd/go-omni/apigen/testdata/compatibility/download.go")}, + } + args.report.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n") + print(f"PASS remote compatibility contract {args.version}; temporary files removed", flush=True) + + +if __name__ == "__main__": + main()