From 20e84e5ff86d872f42301cd3c256030fc41c8ce2 Mon Sep 17 00:00:00 2001 From: k2safe Date: Thu, 24 Sep 2026 02:18:18 +0800 Subject: [PATCH] feat: support database connectors and HTTP server configuration --- CHANGELOG.md | 9 + README.md | 35 ++- RELEASE.md | 44 ++-- app/app.go | 12 +- app/httpConfig_test.go | 199 +++++++++++++++++ app/transport.go | 22 ++ cmd/go-omni/README.md | 4 +- cmd/go-omni/scaffold/version.go | 2 +- config/config.go | 10 + config/env.go | 22 ++ config/http.go | 45 ++++ config/http_test.go | 87 ++++++++ database/README.md | 62 +++++- database/connector.go | 82 +++++++ database/connector_test.go | 336 +++++++++++++++++++++++++++++ database/database.go | 49 +++-- database/mysqlConnector_test.go | 180 ++++++++++++++++ omni.go | 3 + performance/connectorAcceptance.md | 24 +++ performance/connectorLocal.json | 70 ++++++ scripts/verifyConnector.py | 133 ++++++++++++ transport/README.md | 36 +++- 22 files changed, 1403 insertions(+), 63 deletions(-) create mode 100644 app/httpConfig_test.go create mode 100644 config/http.go create mode 100644 config/http_test.go create mode 100644 database/connector.go create mode 100644 database/connector_test.go create mode 100644 database/mysqlConnector_test.go create mode 100644 performance/connectorAcceptance.md create mode 100644 performance/connectorLocal.json create mode 100644 scripts/verifyConnector.py diff --git a/CHANGELOG.md b/CHANGELOG.md index e2afabb..51edd58 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## v0.9.1-preview + +- 新增标准 `database.OpenConnector` 和独立 `PoolConfig`；与 DSN 入口共享池参数、限时启动 Ping、失败清理、Health、事务，不注册全局驱动配置。 +- 新增 `omni.WithHTTPConfig` / `Config.HTTP`：读头、读请求、写响应、空闲超时和头大小，贯通代码、JSON/YAML、环境变量、Run/Serve 与 NewServer；nethttp 支持 Go-only ErrorLog。 +- 保持已有默认行为；负值和实验 http2 不支持的超时项在监听前报错；记录各网络层的具体语义。 +- 增加真实 MySQL TLS/驱动日志隔离、连接池归属/取消/事务、nethttp/native 慢连接与写背压验收，以及无本地 replace 的远程消费者脚本。 +- 兼容 v0.9.0-preview 生成代码；不修改业务项目的数据库策略或生成代码最低版本。 + + ## v0.9.0-preview - 标准 HTTP Handler/Middleware 桥接：跟踪 Raw 状态与提交，保留取消、流式能力和中间件跨请求状态；已提交响应不追加 JSON。 diff --git a/README.md b/README.md index 9ba8c78..7e43534 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ GoOmni 是面向多个业务项目复用的 Go 后端框架：用 `.api` 定义接口、请求和返回类型，生成项目目录、HTTP 接入代码及 OpenAPI，业务代码负责 Service 和 Repository。也可以在已有项目中直接使用 `omni.New()`，或单独引入日志、存储等模块。 -**从这份 README 开始即可接入。** 当前版本为 **`v0.9.0-preview`**，Go **1.22+**；CLI 和项目依赖使用同一固定版本。框架不依赖 Gin，默认使用标准库网络层，提供可切换的 native HTTP/1.1 和实验性 HTTP/2 适配器。 +**从这份 README 开始即可接入。** 当前版本为 **`v0.9.1-preview`**，Go **1.22+**；CLI 和项目依赖使用同一固定版本。框架不依赖 Gin，默认使用标准库网络层，提供可切换的 native HTTP/1.1 和实验性 HTTP/2 适配器。 响应字段、业务码、登录方式、权限规则和数据库表结构由接入项目决定。文中的 `code/msg/data`、`200/9999` 都是示例配置，可以替换。 @@ -26,7 +26,7 @@ module 地址是 `git.yuntuo.hk/k2safe/go-omni`。先确保本机 Go 和 Git 可 ```sh export GOPRIVATE=git.yuntuo.hk -go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.1-preview export PATH="$(go env GOPATH)/bin:$PATH" go-omni --help ``` @@ -38,7 +38,7 @@ go-omni --help 在准备存放业务项目的目录执行，`./hello` 必须是新目录或空目录；把 `example.com/hello` 换成自己的 Go module 名称。 ```sh -go-omni api new --framework-version v0.9.0-preview \ +go-omni api new --framework-version v0.9.1-preview \ --module example.com/hello ./hello cd hello go mod tidy @@ -126,7 +126,7 @@ service catalog-api { ```sh go-omni api validate --api catalog.api -go-omni new --api catalog.api --framework-version v0.9.0-preview \ +go-omni new --api catalog.api --framework-version v0.9.1-preview \ --module example.com/catalog ./catalog cd catalog go mod tidy @@ -167,21 +167,21 @@ curl -i http://127.0.0.1:8080/api/v1/items/missing 需要 MySQL、文件存储或文件日志时，**首次生成**可以使用： ```sh -go-omni new --api catalog.api --framework-version v0.9.0-preview \ +go-omni new --api catalog.api --framework-version v0.9.1-preview \ --module example.com/catalogFull --database mysql --storage local \ --log-output file --log-format json ./catalogFull ``` 该命令在 `catalog.api` 所在目录执行，目标仍需是新目录。数据库迁移、表结构和 Repository 由项目维护，API 模板不自动建表；MySQL DSN 通过 `GO_OMNI_DB_DSN` 提供。存储还可选 `s3`、`r2`、`oss`，按生成项目的配置说明提供桶、端点和凭据。完整装配例子见 [userService](examples/userService/README.md)。 -不采用 `.api` 时，也可以用 `go-omni new --framework-version v0.9.0-preview --module example.com/billing ./billing` 创建手写接口的 CRUD 示例项目；两种生成方式选择一种即可。 +不采用 `.api` 时，也可以用 `go-omni new --framework-version v0.9.1-preview --module example.com/billing ./billing` 创建手写接口的 CRUD 示例项目；两种生成方式选择一种即可。 ## 已有项目：按模块接入 在业务项目的 **go.mod 所在目录**执行： ```sh -go get git.yuntuo.hk/k2safe/go-omni@v0.9.0-preview +go get git.yuntuo.hk/k2safe/go-omni@v0.9.1-preview ``` 已有项目可直接复用现有目录、Service、Repository 和数据库连接，先选择一个接口接入。也可以只用 `logx`、`storage` 等包，暂时保留原来的 HTTP 框架。 @@ -313,6 +313,8 @@ auth UserAuth ( 依赖在启动入口或项目自己的 `internal/servicecontext` 中创建，通过构造函数传给 Service/Repository，避免每个请求都新建连接池。只注册自己拥有的资源，例如 `app.OnShutdown(db.Close)`；关闭按注册的逆序执行。 +已有项目使用 `driver.Connector` 时，直接 `database.OpenConnector(ctx, connector, database.DefaultPoolConfig())`。MySQL 等驱动的独立 TLS、Logger、时区与拨号参数由项目构造 Connector；框架负责连接池、启动 Ping、健康检查及事务。无需转成 DSN，也无需注册全局 TLS/Logger。具体示例和资源归属见 [Connector 接入](database/README.md#connector-接入v091-preview)。 + | 能力 | 包与接入说明 | | --- | --- | | 数据库连接池、事务、显式迁移 | `database`，见 [数据库](database/README.md)；表结构与驱动由项目选择 | @@ -332,6 +334,9 @@ API 项目的数据库迁移需要单独维护并显式执行；启动不会自 | `GO_OMNI_HTTP_ADDR` | 监听地址；生成项目默认 `127.0.0.1:8080` | | `GO_OMNI_TRANSPORT` | `nethttp`（默认）、`native`、`http2` | | `GO_OMNI_TLS_CERT_FILE` / `GO_OMNI_TLS_KEY_FILE` | 成对设置 TLS 证书和私钥路径 | +| `GO_OMNI_HTTP_READ_HEADER_TIMEOUT` / `GO_OMNI_HTTP_READ_TIMEOUT` | 请求头 / 请求读取超时，如 `5s` / `30s` | +| `GO_OMNI_HTTP_WRITE_TIMEOUT` / `GO_OMNI_HTTP_IDLE_TIMEOUT` | 响应写入 / Keep-Alive 空闲超时 | +| `GO_OMNI_HTTP_MAX_HEADER_BYTES` | 请求头大小限制，字节数 | | `GO_OMNI_SHUTDOWN_TIMEOUT` | 优雅退出等待时间，默认 `10s` | | `GO_OMNI_ENABLE_LOGGER` / `GO_OMNI_LOG_LEVEL` | 访问日志开关、日志级别 | | `GO_OMNI_TRUSTED_PROXIES` | 可解析转发地址头的可信代理 IP/CIDR，逗号分隔 | @@ -345,6 +350,20 @@ GO_OMNI_TRANSPORT=native go run ./cmd/server 手写入口可用 `omni.WithTransport(omni.TransportNative)`；TLS 配置为 `omni.WithTLS(certFile, keyFile)`。native 提供 HTTP/1.1、Keep-Alive、TLS、超时、连接/请求并发上限和对象复用；HTTP/2 是单独的实验适配器，必须配置 TLS。默认 nethttp 的 TLS 支持 HTTP/1.1 与 HTTP/2。 +读写超时也可以在生成项目的 `etc/config.yaml` 中声明（放在 `server.http` 下）： + +```yaml +server: + http: + readHeaderTimeout: 5s + readTimeout: 30s + writeTimeout: 30s + idleTimeout: 60s + maxHeaderBytes: 1048576 +``` + +代码配置使用 `omni.WithHTTPConfig(omni.HTTPConfig{ReadTimeout: 30*time.Second, WriteTimeout: 30*time.Second})`。`Run`、`RunContext`、`ServeContext` 会应用它；`NewServer` 会配置返回的标准 `http.Server`，包含可选的 Go 字段 `ErrorLog`。**正值覆盖、零值保留所选网络层的原有默认行为、负值报错**；不配置时升级行为不变。超时含义与支持范围见 [HTTP 配置](transport/README.md#http-配置v091-preview)，不要把网络 I/O 超时当成业务执行时限。 + 自研网络层通过 `RunWith(ctx, runner)` 接入监听与连接生命周期，通过 `NewWithRouter(router)` 替换路由；业务 Handler/Service 不需要导入具体适配器。细节与验收契约见 [网络适配层](transport/README.md)。 `Run` 处理 SIGINT/SIGTERM；`RunContext` 接收调用方控制的退出信号。服务先停止接收新请求，等待活跃请求完成，再清理资源；超时会取消请求并关闭连接，业务代码仍需响应 context 取消。 @@ -364,7 +383,7 @@ go test ./... `Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理，手动修改会报告冲突。Service、Repository、中间件实现、main、配置和依赖装配属于项目，重复生成保留手写内容；契约改变后仍需自行调整业务签名并通过编译。将 `.api`、生成文件和 `.goOmni/apiManifest.json` 一起提交，CI 可运行 `generate --check` 检查是否同步。 -升级时先安装目标版本的 CLI，再在业务 module 中 `go get` **同一版本**；API 项目随后执行上述生成与测试步骤，手写项目只需依赖升级及业务回归。`v0.9.0-preview` 的变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。 +升级时先安装目标版本的 CLI，再在业务 module 中 `go get` **同一版本**；API 项目随后执行上述生成与测试步骤，手写项目只需依赖升级及业务回归。`v0.9.1-preview` 的变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。 ## 按需深入 diff --git a/RELEASE.md b/RELEASE.md index bd2457e..0ff03b9 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,34 +1,31 @@ -# v0.9.0-preview +# v0.9.1-preview -本版补齐已有 HTTP 项目的迁移接入能力：标准 Handler/Middleware 桥接、动态 JSON、路由级请求体限额与手写绑定、自定义标量转换。业务的响应格式、身份、资金事务和数据库实现仍由项目决定。 +本版补齐已有项目接入的两个框架缺口：标准数据库 Connector 注入，以及应用级 HTTP 读写超时配置。TLS、数据库驱动日志、字符集/时区、迁移与业务规则继续由项目决定。 ## 安装与升级 ```sh -export GOPRIVATE=git.yuntuo.hk -go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.0-preview -# 在业务 module 中： -go get git.yuntuo.hk/k2safe/go-omni@v0.9.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.1-preview +# 在各业务 module 中： +go get git.yuntuo.hk/k2safe/go-omni@v0.9.1-preview ``` -将私有域名追加到已有 GOPRIVATE；凭据通过本机 Git 凭据管理器配置。CLI 与项目框架依赖一起升级。进入已生成的项目，再运行 `go-omni api generate --dry-run`、`go-omni api generate`、`go test ./...`。 +私有仓库使用本机 Git 凭据管理器，并将 `git.yuntuo.hk` 追加到已有 `GOPRIVATE`。CLI 和项目依赖使用同一固定版本。v0.9.0-preview 生成代码继续兼容，本次无需为了这两个入口重新生成接口。 -## 本次变化 +## 接入变化 -- `httpx.WrapHandler`、`WrapMiddleware`、`WrapFuncMiddleware` 将现有标准代码接入 App 中间件链。Raw 写入会记录真实状态和提交；已经输出的响应不再追加错误 JSON。下载/流式输出不整体缓存，保留底层支持的 HTTP 可选能力和取消信号。 -- `.api` 的 `any` / `interface{}` 支持动态 JSON。数字保留为 json.Number，外围 DTO 严格校验不变。 -- `@body` 按路由声明限额；内置 App 在全局中间件读取前限制 body。`@binding custom`、`@handlerMode manual` 生成保留的项目实现文件；上传、原文验签、下载可自行处理。`@produces` 描述手写成功输出的媒体类型。 -- `scalar` 显式引用项目 Go 类型，调用标准 JSON/Text 方法；输入和输出各自生成 OpenAPI。普通基础类型不会自动放宽。 +- 将项目已有 `sql.OpenDB(connector)` 的池创建替换为 `database.OpenConnector(ctx, connector, poolConfig)`；不要同时保留两个池。返回对象可用于 Health/InTx；需要标准库指针时取 `.DB`。 +- 项目原有 `mysql.NewConnector`、独立 TLS、driver Logger、驱动读写时限、MySQL 版本/时区/排序规则校验继续保留，配置不进入核心框架。 +- 将原来逐项修改 `http.Server` 的读写/空闲/头大小配置放入 `omni.HTTPConfig`，通过 `WithHTTPConfig` 或 `WithConfig` 传入。Go-only `ErrorLog` 支持 nethttp。 +- 原有 `database.Open(ctx, Config)`、手动组装 `http.Server`、自定义 Runner 均可继续使用。零 HTTP 配置保留所选网络层已有默认行为；正值覆盖，负值报错。 -详见 [迁移扩展指南](MIGRATION.md)、[完整契约](examples/contracts/compatibility.api)、[运行示例](examples/compatibility/main.go) 和 [历史更新](CHANGELOG.md)。 +完整可复制示例：[数据库 Connector](database/README.md#connector-接入v091-preview)、[HTTP 配置及语义](transport/README.md#http-配置v091-preview)。其他迁移能力见 [迁移扩展指南](MIGRATION.md)，从项目接入开始见 [README](README.md)。 -## 兼容与边界 +## 边界 -原有 `.api`、公共响应配置和普通 Handler 可以继续使用。新生成代码最低运行时版本为本版。`Raw()` 现在返回跟踪包装器；直接断言某个具体 writer 类型的代码需改用标准接口/ResponseController。 +独立实验性 `TransportHTTP2` 本次支持 IdleTimeout/MaxHeaderBytes 的配置透传，但不支持读头、读请求和写响应超时，设置这些正值会在监听前明确报错。native 与标准库服务器的超时含义不完全相同，详见上表；网络超时不等于 Handler 执行时间限制。`RunWith` 自定义 Runner 的配置由项目管理。 -标准中间件须保留 incoming request context 并同步调用 next；第三方框架私有的参数读取器仍由项目适配。手写 Handler/绑定 hook 初始返回 501；实现和文档一致性由项目负责，未实现不能上线。切换 manual 回自动模式时须显式迁移并移除对应手写文件，生成器不会删除它。 - -新增能力没有引入 Gin 或 go-zero 依赖。不是完整 goctl 语法兼容层，不代表任何业务项目已完成迁移。HTTP/2 仍为实验功能，本轮没有重新做 Linux 长测、代理验收或云存储账号实网验收。 +Connector 必须遵守驱动 context 协议，资源所有权按标准 `database/sql` 管理。有资源的 Connector 不跨池共享；启动 Ping 失败关闭池，成功后应用负责关闭。没有把 eTax/ePay 的数据库策略写进框架，也未修改这些业务项目或宣称它们已完成迁移。 ## 验证入口 @@ -36,10 +33,13 @@ go get git.yuntuo.hk/k2safe/go-omni@v0.9.0-preview GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=1 -count=1 -timeout=300s ./... go vet ./... go build ./... -python3 scripts/verifyCompatibilityRelease.py --version v0.9.0-preview \ - --report performance/compatibilityRemote.json +python3 scripts/verifyConnector.py --local --report performance/connectorLocal.json +python3 scripts/verifyConnector.py --version v0.9.1-preview --report performance/connectorRemote.json +python3 scripts/verifyCompatibilityRelease.py --version v0.9.1-preview --report performance/connectorCompatibilityRemote.json ``` -测试覆盖旧 Handler 404 与访问日志、提交后错误/panic、标准中间件状态/取消/提前写响应、nethttp/native 实际流式下载与请求体超限、multipart 临时文件清理、自定义标量精度/越界/nullable/集合，以及真实生成项目的验签与重生成保护。远程脚本从全新 module cache 安装 CLI，并检查生成项目的固定版本、校验和以及没有本地 replace。 +Connector 脚本只使用本地独立 MySQL 8.4 容器，限制 1 CPU、768 MiB、128 PID，生成一次性 CA/证书及凭据，结束清理自己的容器/卷/临时文件。测试验证正确 TLS、错误 CA/主机名拒绝、每池时区/日志配置隔离、事务、关闭与失败回收。HTTP 测试使用回环地址验证慢头、慢 body、空闲连接、流式写背压及请求头上限。常规单测不依赖 Docker，显式脚本要求真实 TLS 测试通过，不能跳过。 -本版封版验收通过：40 个包全量 race、独立 userService 回归，以及全新缓存的远程消费者验证。详见 [完整验收记录](performance/compatibilityAcceptance.md)。 +历史发布及验收记录保留在 [CHANGELOG](CHANGELOG.md)、[v0.9.0 接入验收](performance/compatibilityAcceptance.md)。 + +全量 race（40 个测试包）、静态检查、构建及本地隔离验收已通过，详见 [验收记录](performance/connectorAcceptance.md)。 diff --git a/app/app.go b/app/app.go index 0afa2bf..a8352bb 100644 --- a/app/app.go +++ b/app/app.go @@ -156,16 +156,24 @@ func (a *App) Run(address ...string) error { return a.RunContext(ctx, address...) } -// NewServer creates a plain standard HTTP server around the application. +// NewServer creates a plain standard HTTP server with the configured HTTP options. +// Invalid HTTP options panic; RunContext/ServeContext return validation errors. // Use RunContext or ServeContext to apply the configured transport and TLS. func (a *App) NewServer(address string) *http.Server { + if err := a.config.HTTP.Validate(); err != nil { + panic(err) + } if address == "" { address = a.config.Address if address == "" { address = ":8080" } } - return &http.Server{Addr: address, Handler: a.router} + cfg := a.config.HTTP + return &http.Server{Addr: address, Handler: a.router, + ReadHeaderTimeout: cfg.ReadHeaderTimeout, ReadTimeout: cfg.ReadTimeout, + WriteTimeout: cfg.WriteTimeout, IdleTimeout: cfg.IdleTimeout, + MaxHeaderBytes: cfg.MaxHeaderBytes, ErrorLog: cfg.ErrorLog} } // RunContext serves until the context is canceled, then performs a graceful shutdown. diff --git a/app/httpConfig_test.go b/app/httpConfig_test.go new file mode 100644 index 0000000..4db06a3 --- /dev/null +++ b/app/httpConfig_test.go @@ -0,0 +1,199 @@ +package app_test + +import ( + "bufio" + "context" + "errors" + "io" + "log" + "net" + "net/http" + "strings" + "sync/atomic" + "testing" + "time" + + omni "git.yuntuo.hk/k2safe/go-omni" +) + +func TestHTTPConfiguredStandardServer(t *testing.T) { + cfg := omni.HTTPConfig{ReadHeaderTimeout: time.Second, ReadTimeout: 2 * time.Second, WriteTimeout: 3 * time.Second, IdleTimeout: 4 * time.Second, MaxHeaderBytes: 8192, ErrorLog: log.New(io.Discard, "", 0)} + a := omni.New(omni.WithHTTPConfig(cfg)) + s := a.NewServer(":9000") + if s.ReadHeaderTimeout != cfg.ReadHeaderTimeout || s.ReadTimeout != cfg.ReadTimeout || s.WriteTimeout != cfg.WriteTimeout || s.IdleTimeout != cfg.IdleTimeout || s.MaxHeaderBytes != cfg.MaxHeaderBytes || s.ErrorLog != cfg.ErrorLog || s.Addr != ":9000" { + t.Fatalf("server options: %+v", s) + } + s = omni.New().NewServer("") + if s.ReadHeaderTimeout != 0 || s.ReadTimeout != 0 || s.WriteTimeout != 0 || s.IdleTimeout != 0 || s.MaxHeaderBytes != 0 || s.ErrorLog != nil { + t.Fatal("default behavior changed") + } +} + +func TestHTTPInvalidStartupReleasesResources(t *testing.T) { + a := omni.New(omni.WithHTTPConfig(omni.HTTPConfig{ReadTimeout: -time.Second})) + called := 0 + cleanupErr := errors.New("cleanup") + a.OnShutdown(func() error { called++; return cleanupErr }) + err := a.RunContext(context.Background(), "bad address") + if err == nil || !strings.Contains(err.Error(), "http.readTimeout") || !errors.Is(err, cleanupErr) || called != 1 { + t.Fatalf("validation/cleanup: %v, %d", err, called) + } + defer func() { + if recover() == nil { + t.Error("NewServer accepted invalid config") + } + }() + a.NewServer("") +} + +type httpConfigListener struct{ net.Listener } + +func (l httpConfigListener) Accept() (net.Conn, error) { + c, err := l.Listener.Accept() + if err == nil { + if tcp, ok := c.(*net.TCPConn); ok { + _ = tcp.SetWriteBuffer(32 << 10) + } + } + return c, err +} + +func httpConfigConnection(t *testing.T, transport string, cfg omni.HTTPConfig, handler omni.Handler) net.Conn { + t.Helper() + a := omni.New(omni.WithTransport(transport), omni.WithHTTPConfig(cfg), omni.WithShutdownTimeout(time.Second)) + a.GET("/", handler) + a.POST("/", handler) + l, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { done <- a.ServeContext(ctx, httpConfigListener{l}) }() + t.Cleanup(func() { + cancel() + select { + case err := <-done: + if err != nil { + t.Errorf("server shutdown: %v", err) + } + case <-time.After(3 * time.Second): + t.Error("server did not stop") + } + }) + c, err := net.DialTimeout("tcp", l.Addr().String(), time.Second) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { c.Close() }) + _ = c.SetDeadline(time.Now().Add(3 * time.Second)) + return c +} + +func TestHTTPConfiguredNetworkLimits(t *testing.T) { + for _, transport := range []string{omni.TransportNetHTTP, omni.TransportNative} { + t.Run(transport, func(t *testing.T) { + empty := func(c *omni.Context) error { w := c.Raw(); w.WriteHeader(204); return nil } + t.Run("slowHeaders", func(t *testing.T) { + c := httpConfigConnection(t, transport, omni.HTTPConfig{ReadHeaderTimeout: 60 * time.Millisecond}, empty) + if _, err := io.WriteString(c, "GET / HTTP/1.1\r\nHost: localhost\r\nX-Incomplete:"); err != nil { + t.Fatal(err) + } + _, err := io.ReadAll(c) + if e, ok := err.(net.Error); ok && e.Timeout() { + t.Fatal("server read-header timeout not applied") + } + }) + t.Run("slowBody", func(t *testing.T) { + var accepted atomic.Bool + c := httpConfigConnection(t, transport, omni.HTTPConfig{ReadTimeout: 60 * time.Millisecond}, func(c *omni.Context) error { + _, err := io.ReadAll(c.Request().Body) + accepted.Store(err == nil) + return nil + }) + if _, err := io.WriteString(c, "POST / HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\nContent-Length: 10\r\n\r\nx"); err != nil { + t.Fatal(err) + } + _, err := io.ReadAll(c) + if e, ok := err.(net.Error); ok && e.Timeout() { + t.Fatal("read timeout not applied") + } + if accepted.Load() { + t.Fatal("incomplete body accepted") + } + }) + t.Run("idle", func(t *testing.T) { + c := httpConfigConnection(t, transport, omni.HTTPConfig{IdleTimeout: 60 * time.Millisecond}, empty) + if _, err := io.WriteString(c, "GET / HTTP/1.1\r\nHost: localhost\r\n\r\n"); err != nil { + t.Fatal(err) + } + r, err := http.ReadResponse(bufio.NewReader(c), nil) + if err != nil { + t.Fatal(err) + } + r.Body.Close() + if r.StatusCode != 204 { + t.Fatal(r.StatusCode) + } + _, err = c.Read(make([]byte, 1)) + if e, ok := err.(net.Error); ok && e.Timeout() { + t.Fatal("idle timeout not applied") + } + if err == nil { + t.Fatal("idle connection not closed") + } + }) + t.Run("writeBackpressure", func(t *testing.T) { + done := make(chan error, 1) + c := httpConfigConnection(t, transport, omni.HTTPConfig{WriteTimeout: 60 * time.Millisecond}, func(c *omni.Context) error { + w := c.Raw() + w.WriteHeader(200) + if err := http.NewResponseController(w).Flush(); err != nil { + done <- err + return nil + } + _, err := io.Copy(w, io.LimitReader(repeatingHTTPReader{}, 64<<20)) + done <- err + return nil + }) + if tcp, ok := c.(*net.TCPConn); ok { + _ = tcp.SetReadBuffer(1024) + } + if _, err := io.WriteString(c, "GET / HTTP/1.1\r\nHost: localhost\r\n\r\n"); err != nil { + t.Fatal(err) + } + select { + case err := <-done: + if err == nil { + t.Fatal("write unexpectedly completed") + } + case <-time.After(2 * time.Second): + t.Fatal("write timeout not applied") + } + }) + t.Run("headerLimit", func(t *testing.T) { + c := httpConfigConnection(t, transport, omni.HTTPConfig{MaxHeaderBytes: 1024}, empty) + if _, err := io.WriteString(c, "GET / HTTP/1.1\r\nHost: localhost\r\nX-Large: "+strings.Repeat("a", 8192)+"\r\n\r\n"); err != nil { + t.Fatal(err) + } + r, err := http.ReadResponse(bufio.NewReader(c), nil) + if err != nil { + t.Fatal(err) + } + r.Body.Close() + if r.StatusCode != 431 { + t.Fatalf("oversized header status: %d", r.StatusCode) + } + }) + }) + } +} + +type repeatingHTTPReader struct{} + +func (repeatingHTTPReader) Read(b []byte) (int, error) { + for i := range b { + b[i] = 'x' + } + return len(b), nil +} diff --git a/app/transport.go b/app/transport.go index f6e6a64..abe1e84 100644 --- a/app/transport.go +++ b/app/transport.go @@ -57,6 +57,22 @@ func (a *App) prepareTransport() (func(context.Context, net.Listener) error, err case config.TransportNative: server := native.New(a) server.ShutdownTimeout = shutdownTimeout + cfg := a.config.HTTP + if cfg.ReadHeaderTimeout > 0 { + server.ReadHeaderTimeout = cfg.ReadHeaderTimeout + } + if cfg.ReadTimeout > 0 { + server.ReadTimeout = cfg.ReadTimeout + } + if cfg.WriteTimeout > 0 { + server.WriteTimeout = cfg.WriteTimeout + } + if cfg.IdleTimeout > 0 { + server.IdleTimeout = cfg.IdleTimeout + } + if cfg.MaxHeaderBytes > 0 { + server.MaxHeaderBytes = cfg.MaxHeaderBytes + } if tlsConfig != nil { return func(ctx context.Context, listener net.Listener) error { return server.ServeTLS(ctx, listener, tlsConfig) @@ -66,6 +82,12 @@ func (a *App) prepareTransport() (func(context.Context, net.Listener) error, err case config.TransportHTTP2: server := h2transport.New(a) server.ShutdownTimeout = shutdownTimeout + if a.config.HTTP.IdleTimeout > 0 { + server.IdleTimeout = a.config.HTTP.IdleTimeout + } + if a.config.HTTP.MaxHeaderBytes > 0 { + server.MaxHeaderBytes = a.config.HTTP.MaxHeaderBytes + } return func(ctx context.Context, listener net.Listener) error { return server.ServeTLS(ctx, listener, tlsConfig) }, nil diff --git a/cmd/go-omni/README.md b/cmd/go-omni/README.md index 62834ae..8a74a15 100644 --- a/cmd/go-omni/README.md +++ b/cmd/go-omni/README.md @@ -75,8 +75,8 @@ Handler / Service / Repository / Model 与模块注册入口。模块名为 lowe 从远程安装，生成一个带 Handler / Service / Repository / Model 分层的服务骨架： ```sh -GOPRIVATE=git.yuntuo.hk go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.0-preview -go-omni new --framework-version v0.9.0-preview --module example.com/billing ./billing +GOPRIVATE=git.yuntuo.hk go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.1-preview +go-omni new --framework-version v0.9.1-preview --module example.com/billing ./billing cd billing go mod tidy go run ./cmd/server diff --git a/cmd/go-omni/scaffold/version.go b/cmd/go-omni/scaffold/version.go index 67da8b3..75a2b02 100644 --- a/cmd/go-omni/scaffold/version.go +++ b/cmd/go-omni/scaffold/version.go @@ -10,7 +10,7 @@ const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" // Source builds can carry an unpublished VCS version (including +dirty). // Keep this fallback in sync with the next release; installed CLIs use their // downloaded module version, which also has a module checksum. -const defaultFrameworkVersion = "v0.9.0-preview" +const defaultFrameworkVersion = "v0.9.1-preview" // This target is not a claim that the release already exists. Source users // select a local framework checkout until the module templates are published. diff --git a/config/config.go b/config/config.go index 63eade8..8081d55 100644 --- a/config/config.go +++ b/config/config.go @@ -28,6 +28,7 @@ type Config struct { Address string `json:"address"` Transport string `json:"transport"` TLS TLSConfig `json:"tls"` + HTTP HTTPConfig `json:"http"` RequestIDHeader string `json:"requestIdHeader"` EnableLogger bool `json:"enableLogger"` SlowRequestThreshold time.Duration `json:"slowRequestThreshold"` @@ -74,11 +75,20 @@ func (c Config) Validate() error { // ValidateTransport checks built-in transport selection without opening files // or sockets. Empty Transport preserves the standard net/http default. func (c Config) ValidateTransport() error { + if err := c.HTTP.Validate(); err != nil { + return err + } switch c.Transport { case "", TransportNetHTTP, TransportNative, TransportHTTP2: default: return &ConfigError{Field: "transport", Message: "must be nethttp, native or http2"} } + if c.Transport == TransportHTTP2 && (c.HTTP.ReadHeaderTimeout > 0 || c.HTTP.ReadTimeout > 0 || c.HTTP.WriteTimeout > 0) { + return &ConfigError{Field: "http", Message: "readHeaderTimeout, readTimeout and writeTimeout are not supported by the experimental http2 transport"} + } + if c.HTTP.ErrorLog != nil && c.Transport != "" && c.Transport != TransportNetHTTP { + return &ConfigError{Field: "http.errorLog", Message: "is supported only by the nethttp transport"} + } cert, key := strings.TrimSpace(c.TLS.CertFile), strings.TrimSpace(c.TLS.KeyFile) if (cert == "") != (key == "") { return &ConfigError{Field: "tls", Message: "requires both certFile and keyFile"} diff --git a/config/env.go b/config/env.go index 8eb3782..d777886 100644 --- a/config/env.go +++ b/config/env.go @@ -48,6 +48,28 @@ func applyEnv(cfg Config, prefix string) (Config, error) { cfg.Log.Environment = envString(prefix+"ENVIRONMENT", cfg.Log.Environment) var err error + for _, field := range []struct { + name string + value *time.Duration + }{ + {"HTTP_READ_HEADER_TIMEOUT", &cfg.HTTP.ReadHeaderTimeout}, + {"HTTP_READ_TIMEOUT", &cfg.HTTP.ReadTimeout}, + {"HTTP_WRITE_TIMEOUT", &cfg.HTTP.WriteTimeout}, + {"HTTP_IDLE_TIMEOUT", &cfg.HTTP.IdleTimeout}, + } { + if value := strings.TrimSpace(os.Getenv(prefix + field.name)); value != "" { + *field.value, err = time.ParseDuration(value) + if err != nil { + return Config{}, fmt.Errorf("%s%s must be a duration: %w", prefix, field.name, err) + } + } + } + if value := strings.TrimSpace(os.Getenv(prefix + "HTTP_MAX_HEADER_BYTES")); value != "" { + cfg.HTTP.MaxHeaderBytes, err = strconv.Atoi(value) + if err != nil { + return Config{}, fmt.Errorf("%sHTTP_MAX_HEADER_BYTES must be an integer: %w", prefix, err) + } + } if cfg.EnableLogger, err = envBool(prefix+"ENABLE_LOGGER", cfg.EnableLogger); err != nil { return Config{}, err } diff --git a/config/http.go b/config/http.go new file mode 100644 index 0000000..1d7b460 --- /dev/null +++ b/config/http.go @@ -0,0 +1,45 @@ +package config + +import ( + "log" + "time" +) + +// HTTPConfig controls built-in HTTP servers. Positive values override the +// selected transport's defaults; zero keeps its existing behavior. Timeouts +// follow that transport's semantics, not a handler execution deadline. +type HTTPConfig struct { + ReadHeaderTimeout time.Duration `json:"readHeaderTimeout"` + ReadTimeout time.Duration `json:"readTimeout"` + WriteTimeout time.Duration `json:"writeTimeout"` + IdleTimeout time.Duration `json:"idleTimeout"` + MaxHeaderBytes int `json:"maxHeaderBytes"` + // ErrorLog is a Go-only hook for the nethttp server's internal diagnostics. + ErrorLog *log.Logger `json:"-"` +} + +func (c HTTPConfig) Validate() error { + for _, field := range []struct { + name string + value time.Duration + }{ + {"readHeaderTimeout", c.ReadHeaderTimeout}, + {"readTimeout", c.ReadTimeout}, + {"writeTimeout", c.WriteTimeout}, + {"idleTimeout", c.IdleTimeout}, + } { + if field.value < 0 { + return &ConfigError{Field: "http." + field.name, Message: "must be nonnegative"} + } + } + if c.MaxHeaderBytes < 0 { + return &ConfigError{Field: "http.maxHeaderBytes", Message: "must be nonnegative"} + } + return nil +} + +// WithHTTPConfig sets the built-in server's HTTP options. RunWith uses a +// caller-owned Runner and does not apply these options to that runner. +func WithHTTPConfig(settings HTTPConfig) Option { + return func(cfg *Config) { cfg.HTTP = settings } +} diff --git a/config/http_test.go b/config/http_test.go new file mode 100644 index 0000000..b848cf2 --- /dev/null +++ b/config/http_test.go @@ -0,0 +1,87 @@ +package config_test + +import ( + "io" + "log" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "git.yuntuo.hk/k2safe/go-omni/config" +) + +func TestHTTPFileAndEnvironment(t *testing.T) { + for _, file := range []struct{ name, body string }{ + {"server.json", `{"http":{"readHeaderTimeout":"2s","readTimeout":"3s","writeTimeout":"4s","idleTimeout":"5s","maxHeaderBytes":8192}}`}, + {"server.yaml", "http:\n readHeaderTimeout: 2s\n readTimeout: 3s\n writeTimeout: 4s\n idleTimeout: 5s\n maxHeaderBytes: 8192\n"}, + } { + t.Run(file.name, func(t *testing.T) { + path := filepath.Join(t.TempDir(), file.name) + if err := os.WriteFile(path, []byte(file.body), 0600); err != nil { + t.Fatal(err) + } + cfg, err := config.LoadFile(path, "HTTP_TEST_") + if err != nil { + t.Fatal(err) + } + want := config.HTTPConfig{ReadHeaderTimeout: 2 * time.Second, ReadTimeout: 3 * time.Second, WriteTimeout: 4 * time.Second, IdleTimeout: 5 * time.Second, MaxHeaderBytes: 8192} + if cfg.HTTP != want { + t.Fatalf("HTTP: %+v", cfg.HTTP) + } + t.Setenv("HTTP_TEST_HTTP_READ_HEADER_TIMEOUT", "6s") + t.Setenv("HTTP_TEST_HTTP_READ_TIMEOUT", "7s") + t.Setenv("HTTP_TEST_HTTP_WRITE_TIMEOUT", "8s") + t.Setenv("HTTP_TEST_HTTP_IDLE_TIMEOUT", "9s") + t.Setenv("HTTP_TEST_HTTP_MAX_HEADER_BYTES", "16384") + got, err := config.ApplyEnv(cfg, "HTTP_TEST_") + if err != nil { + t.Fatal(err) + } + want = config.HTTPConfig{ReadHeaderTimeout: 6 * time.Second, ReadTimeout: 7 * time.Second, WriteTimeout: 8 * time.Second, IdleTimeout: 9 * time.Second, MaxHeaderBytes: 16384} + if got.HTTP != want { + t.Fatalf("overlay: %+v", got.HTTP) + } + }) + } +} + +func TestHTTPInvalidAndUnsupportedSettings(t *testing.T) { + for _, key := range []string{"READ_HEADER_TIMEOUT", "READ_TIMEOUT", "WRITE_TIMEOUT", "IDLE_TIMEOUT", "MAX_HEADER_BYTES"} { + for _, value := range []string{"invalid", "-1"} { + t.Run(key+value, func(t *testing.T) { + if value == "-1" && key != "MAX_HEADER_BYTES" { + value = "-1s" + } + t.Setenv("HTTP_INVALID_HTTP_"+key, value) + if _, err := config.LoadFromEnv("HTTP_INVALID_"); err == nil { + t.Fatal("invalid HTTP value accepted") + } + }) + } + } + for _, http := range []config.HTTPConfig{{ReadHeaderTimeout: time.Second}, {ReadTimeout: time.Second}, {WriteTimeout: time.Second}} { + cfg := config.Default() + cfg.Transport = config.TransportHTTP2 + cfg.TLS = config.TLSConfig{CertFile: "c", KeyFile: "k"} + cfg.HTTP = http + if err := cfg.ValidateTransport(); err == nil || !strings.Contains(err.Error(), "not supported") { + t.Fatalf("HTTP/2: %v", err) + } + } + cfg := config.Default() + cfg.Transport = config.TransportHTTP2 + cfg.TLS = config.TLSConfig{CertFile: "c", KeyFile: "k"} + cfg.HTTP = config.HTTPConfig{IdleTimeout: time.Minute, MaxHeaderBytes: 8192} + if err := cfg.ValidateTransport(); err != nil { + t.Fatal(err) + } + for _, transport := range []string{config.TransportNative, config.TransportHTTP2} { + cfg.Transport = transport + cfg.HTTP = config.HTTPConfig{ErrorLog: log.New(io.Discard, "", 0)} + if err := cfg.ValidateTransport(); err == nil || !strings.Contains(err.Error(), "http.errorLog") { + t.Fatalf("ErrorLog silently ignored: %v", err) + } + } +} diff --git a/database/README.md b/database/README.md index 7a7b26d..a24aba4 100644 --- a/database/README.md +++ b/database/README.md @@ -3,7 +3,7 @@ `database` 包提供基于标准库 `database/sql` 的连接池生命周期，不绑定 ORM 或驱动： - `Config` / `ConfigFromEnv`：连接池参数和 `GO_OMNI_DB_*` 环境变量。 -- `Open`：创建连接池、设置连接上限并在返回前执行 `PingContext`。 +- `Open` / `OpenConnector`：通过 DSN 或标准 Connector 创建连接池、设置连接上限并在返回前执行 `PingContext`。 - `Health`：用于就绪检查，等待连接和 Ping 均受 `PingTimeout` 限制。 - `InTx`：统一事务提交和回滚；工作函数返回错误或 panic 时回滚。 - `Close`：由嵌入的 `*sql.DB` 提供，应用退出时显式调用。 @@ -40,7 +40,7 @@ GO_OMNI_DB_CONN_MAX_IDLE_TIME=5m GO_OMNI_DB_PING_TIMEOUT=5s ``` -`DSN` 必填，驱动默认是 `mysql`。数据库建库、迁移、表结构和具体 Repository 属于业务项目或可选模块，不会被核心包自动执行。 +使用 `ConfigFromEnv` / `Open` 时 `DSN` 必填，环境变量入口的驱动默认是 `mysql`。数据库建库、迁移、表结构和具体 Repository 属于业务项目或可选模块，不会被核心包自动执行。 `MaxIdleConns=0` 会禁用空闲连接。初始化 Ping 失败会关闭已创建的连接池。 @@ -64,3 +64,61 @@ python3 scripts/verifyDatabase.py --version v0.2.0 `Config` 的 JSON/YAML 字段使用 lowerCamelCase，`DSN` 使用 `json:"-"`，连接密钥从环境变量加载。 API 脚手架 `--database mysql` 使用此流程，并要求正的 `maxOpenConns`，提供数据库 readiness 与幂等清理。 具体表结构、迁移和 Repository 留在业务项目，参考 `examples/userService`。 + +## Connector 接入（v0.9.1-preview） + +项目已有 `database/sql/driver.Connector` 时使用 `OpenConnector`。这是标准驱动接口，核心包不增加具体驱动依赖。项目可以在同一进程创建多个使用不同证书、日志、拨号策略的连接池。 + +下面是 MySQL 的完整组装函数；连接地址、凭据、TLS 和驱动 Logger 都从调用方传入，不在框架中约定： + +```go +package connection + +import ( + "context" + "crypto/tls" + "github.com/go-sql-driver/mysql" + "git.yuntuo.hk/k2safe/go-omni/database" +) + +func Open(ctx context.Context, address, user, password, name string, + tlsConfig *tls.Config, logger mysql.Logger, + pool database.PoolConfig) (*database.DB, error) { + cfg := mysql.NewConfig() + cfg.Net, cfg.Addr = "tcp", address + cfg.User, cfg.Passwd, cfg.DBName = user, password, name + cfg.TLS = tlsConfig // 项目设置 RootCAs、ServerName、MinVersion 等 + if logger != nil { + cfg.Logger = logger // 项目现有的 Print(...any) 日志适配器 + } + connector, err := mysql.NewConnector(cfg) + if err != nil { + return nil, err + } + return database.OpenConnector(ctx, connector, pool) +} +``` + +调用时以 `database.DefaultPoolConfig()` 为起点，按项目需要调整。已有 `database.Config` 可调用 `.PoolConfig()` 提取池设置；`ConfigFromEnv` / `ApplyEnv` 仍是 DSN 入口，不用为 Connector 填假 DSN。`PoolConfig` 支持通过 `config.DecodeFile` 加载项目的 JSON/YAML 嵌套配置，环境覆盖由项目按自己的驱动配置组织。 + +| PoolConfig 字段 | 默认值 / 含义 | +| --- | --- | +| `MaxOpenConns` | `10`；`0` 表示不限制 | +| `MaxIdleConns` | `5`；`0` 禁止保留空闲连接 | +| `ConnMaxLifetime` | `30m`；`0` 不因连接年龄回收 | +| `ConnMaxIdleTime` | `5m`；`0` 不因空闲时长回收 | +| `PingTimeout` | `5s`；必须为正，约束启动 Ping 和 Health 等待 | + +返回的 `*database.DB` 嵌入 `*sql.DB`，可使用 `QueryContext`、`ExecContext`、`Stats`，以及框架的 `Health`、`InTx`。需要标准数据库指针的 ORM/迁移器可取 `db.DB`；不要重复建立池。项目负责驱动支持的 context 取消与读写超时设置，框架不能强制中断不遵守 context 的自定义驱动。 + +**生命周期：** nil/带类型的 nil Connector、非法池参数、启动前已取消的 context 会在创建池前返回，Connector 仍归调用方。进入 `sql.OpenDB` 后由池持有；启动 Ping 失败自动关闭，成功后由调用方负责 `db.Close`，通常注册 `app.OnShutdown(db.Close)`。若 Connector 实现 `io.Closer`，遵循标准 `database/sql` 规则随池关闭；有资源的 Connector 不应跨多个池共享。资源初始化过程中、注册退出回调之前若业务校验失败，项目也须关闭池。 + +`db.PoolConfig()` 返回池配置快照。Connector 入口的 `db.Config()` 中 `Driver` / `DSN` 为空，不能用它重建 Connector。TLS、Logger、驱动版本检查、字符集、时区以及迁移规则保持由项目负责；不写入框架配置快照。 + +验收脚本使用独立本地 MySQL 8.4 容器（1 CPU / 768 MiB），自动生成测试 CA/证书与随机凭据，检查 TLS 信任/主机名、两池配置隔离、驱动日志、事务和 HTTP 超时，结束后清理本次资源： + +```sh +python3 scripts/verifyConnector.py --local --report performance/connectorLocal.json +# 发布后通过全新 module cache 验证远程依赖，无本地 replace： +python3 scripts/verifyConnector.py --version v0.9.1-preview --report performance/connectorRemote.json +``` diff --git a/database/connector.go b/database/connector.go new file mode 100644 index 0000000..89704be --- /dev/null +++ b/database/connector.go @@ -0,0 +1,82 @@ +package database + +import ( + "context" + "database/sql" + "database/sql/driver" + "errors" + "reflect" + "time" +) + +// PoolConfig contains only driver-independent settings. Connection credentials, +// TLS, logging and dial behavior belong to the application's Connector. +// Start from DefaultPoolConfig; zero PingTimeout is invalid. +type PoolConfig struct { + MaxOpenConns int `json:"maxOpenConns"` + MaxIdleConns int `json:"maxIdleConns"` + ConnMaxLifetime time.Duration `json:"connMaxLifetime"` + ConnMaxIdleTime time.Duration `json:"connMaxIdleTime"` + PingTimeout time.Duration `json:"pingTimeout"` +} + +func DefaultPoolConfig() PoolConfig { + return PoolConfig{MaxOpenConns: 10, MaxIdleConns: 5, ConnMaxLifetime: 30 * time.Minute, ConnMaxIdleTime: 5 * time.Minute, PingTimeout: 5 * time.Second} +} + +func (c PoolConfig) Validate() error { + if c.MaxOpenConns < 0 || c.MaxIdleConns < 0 { + return errors.New("database pool limits cannot be negative") + } + if c.MaxOpenConns > 0 && c.MaxIdleConns > c.MaxOpenConns { + return errors.New("max idle connections cannot exceed max open connections") + } + if c.ConnMaxLifetime < 0 || c.ConnMaxIdleTime < 0 || c.PingTimeout <= 0 { + return errors.New("database timeouts are invalid") + } + return nil +} + +func (c PoolConfig) config() Config { + return Config{MaxOpenConns: c.MaxOpenConns, MaxIdleConns: c.MaxIdleConns, ConnMaxLifetime: c.ConnMaxLifetime, ConnMaxIdleTime: c.ConnMaxIdleTime, PingTimeout: c.PingTimeout} +} + +// OpenConnector opens an independent pool using the supplied standard Connector. +// No registered driver name or DSN is required. It applies cfg and completes a +// startup Ping bounded by ctx and PingTimeout before returning. +// +// Validation failures and an already-canceled ctx leave the Connector untouched. +// Once sql.OpenDB is called, the pool owns its lifecycle: a failed startup closes +// the pool; on success the caller must Close the returned DB. As in database/sql, +// this also closes a Connector that implements io.Closer. Use a separate Connector +// for each pool if it owns resources. Nil ctx means context.Background. +func OpenConnector(ctx context.Context, connector driver.Connector, cfg PoolConfig) (*DB, error) { + if nilConnector(connector) { + return nil, errors.New("database connector is required") + } + if err := cfg.Validate(); err != nil { + return nil, err + } + if ctx == nil { + ctx = context.Background() + } + if err := ctx.Err(); err != nil { + return nil, err + } + return initializePool(ctx, sql.OpenDB(connector), cfg.config()) +} + +func nilConnector(connector driver.Connector) bool { + if connector == nil { + return true + } + v := reflect.ValueOf(connector) + switch v.Kind() { + case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice: + return v.IsNil() + } + return false +} + +// PoolConfig returns a copy of the settings used to create this pool. +func (db *DB) PoolConfig() PoolConfig { return db.config.PoolConfig() } diff --git a/database/connector_test.go b/database/connector_test.go new file mode 100644 index 0000000..6f650f3 --- /dev/null +++ b/database/connector_test.go @@ -0,0 +1,336 @@ +package database_test + +import ( + "context" + "database/sql" + "database/sql/driver" + "errors" + "io" + "sync" + "sync/atomic" + "testing" + "time" + + "git.yuntuo.hk/k2safe/go-omni/database" +) + +type fixtureConnector struct { + identity string + connects, closedConnections, closedConnector, commits, rollbacks atomic.Int32 + connect func(context.Context) error + ping func(context.Context) error + query func(context.Context) error + closeErr error +} + +func (c *fixtureConnector) Connect(ctx context.Context) (driver.Conn, error) { + c.connects.Add(1) + if c.connect != nil { + if err := c.connect(ctx); err != nil { + return nil, err + } + } + return &fixtureConn{owner: c}, nil +} +func (c *fixtureConnector) Driver() driver.Driver { return fixtureDriver{} } +func (c *fixtureConnector) Close() error { c.closedConnector.Add(1); return c.closeErr } + +type fixtureDriver struct{} + +func (fixtureDriver) Open(string) (driver.Conn, error) { + return nil, errors.New("unexpected driver name/DSN path") +} + +type fixtureConn struct{ owner *fixtureConnector } + +func (c *fixtureConn) Prepare(string) (driver.Stmt, error) { return nil, errors.New("not used") } +func (c *fixtureConn) Close() error { c.owner.closedConnections.Add(1); return nil } +func (c *fixtureConn) Begin() (driver.Tx, error) { return fixtureTx{c.owner}, nil } +func (c *fixtureConn) Ping(ctx context.Context) error { + if c.owner.ping != nil { + return c.owner.ping(ctx) + } + return nil +} +func (c *fixtureConn) QueryContext(ctx context.Context, _ string, _ []driver.NamedValue) (driver.Rows, error) { + if c.owner.query != nil { + if err := c.owner.query(ctx); err != nil { + return nil, err + } + } + return &fixtureRows{identity: c.owner.identity}, nil +} + +type fixtureTx struct{ owner *fixtureConnector } + +func (tx fixtureTx) Commit() error { tx.owner.commits.Add(1); return nil } +func (tx fixtureTx) Rollback() error { tx.owner.rollbacks.Add(1); return nil } + +type fixtureRows struct { + identity string + sent bool +} + +func (r *fixtureRows) Columns() []string { return []string{"identity"} } +func (r *fixtureRows) Close() error { return nil } +func (r *fixtureRows) Next(values []driver.Value) error { + if r.sent { + return io.EOF + } + r.sent = true + values[0] = r.identity + return nil +} + +func TestConnectorIndependentPoolsAndContext(t *testing.T) { + type key struct{} + ctx := context.WithValue(context.Background(), key{}, "startup") + cfg := database.DefaultPoolConfig() + cfg.MaxOpenConns = 2 + cfg.MaxIdleConns = 2 + var owners []*fixtureConnector + var pools []*database.DB + for _, identity := range []string{"alpha", "beta"} { + owner := &fixtureConnector{identity: identity} + // Startup Connect/Ping and each query preserve their caller context. + // database/sql may also open replacement connections in the background. + owner.connect = func(ctx context.Context) error { + if owner.connects.Load() == 1 && ctx.Value(key{}) != "startup" { + return errors.New("lost request context") + } + return nil + } + owner.ping = func(ctx context.Context) error { + if ctx.Value(key{}) != "startup" { + return errors.New("lost ping context") + } + return nil + } + owner.query = func(ctx context.Context) error { + if ctx.Value(key{}) != "startup" { + return errors.New("lost query context") + } + return nil + } + pool, err := database.OpenConnector(ctx, owner, cfg) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { pool.Close() }) + if pool.PoolConfig() != cfg || pool.Config().Driver != "" || pool.Config().DSN != "" || pool.Stats().MaxOpenConnections != 2 { + t.Fatal("pool settings or DSN boundary") + } + owners = append(owners, owner) + pools = append(pools, pool) + } + var wg sync.WaitGroup + for i, pool := range pools { + for n := 0; n < 16; n++ { + wg.Add(1) + go func(i int, pool *database.DB) { + defer wg.Done() + var identity string + if err := pool.QueryRowContext(ctx, "identity").Scan(&identity); err != nil { + t.Error(err) + } else if identity != owners[i].identity { + t.Error("crossed connectors") + } + }(i, pool) + } + } + wg.Wait() + for i, pool := range pools { + if pool.Stats().OpenConnections > cfg.MaxOpenConns { + t.Fatal("pool cap not applied") + } + if err := pool.Close(); err != nil { + t.Fatal(err) + } + if err := pool.Close(); err != nil { + t.Fatal(err) + } + if owners[i].closedConnector.Load() != 1 || owners[i].closedConnections.Load() != owners[i].connects.Load() { + t.Fatal("close did not release owned resources exactly once") + } + if pool.Health(ctx) == nil { + t.Fatal("closed pool is healthy") + } + } +} + +func TestConnectorRejectsInvalidInputWithoutTakingOwnership(t *testing.T) { + var typedNil *fixtureConnector + for _, connector := range []driver.Connector{nil, typedNil} { + if db, err := database.OpenConnector(nil, connector, database.DefaultPoolConfig()); err == nil || db != nil { + t.Fatal("nil connector accepted") + } + } + for _, mutate := range []func(*database.PoolConfig){ + func(c *database.PoolConfig) { c.MaxOpenConns = -1 }, func(c *database.PoolConfig) { c.MaxIdleConns = -1 }, + func(c *database.PoolConfig) { c.MaxIdleConns = c.MaxOpenConns + 1 }, func(c *database.PoolConfig) { c.PingTimeout = 0 }, + func(c *database.PoolConfig) { c.ConnMaxLifetime = -1 }, func(c *database.PoolConfig) { c.ConnMaxIdleTime = -1 }, + } { + cfg := database.DefaultPoolConfig() + mutate(&cfg) + owner := &fixtureConnector{} + if db, err := database.OpenConnector(nil, owner, cfg); err == nil || db != nil { + t.Fatal("invalid config accepted") + } + if owner.connects.Load() != 0 || owner.closedConnector.Load() != 0 { + t.Fatal("validation consumed connector") + } + } + owner := &fixtureConnector{} + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if db, err := database.OpenConnector(ctx, owner, database.DefaultPoolConfig()); !errors.Is(err, context.Canceled) || db != nil { + t.Fatalf("pre-canceled context: %v", err) + } + if owner.connects.Load() != 0 || owner.closedConnector.Load() != 0 { + t.Fatal("pre-cancellation consumed connector") + } + if err := database.DefaultConfig().Validate(); err == nil { + t.Fatal("legacy DSN requirements changed") + } +} + +func TestConnectorStartupFailuresReleaseResources(t *testing.T) { + failure := errors.New("fixture unavailable") + closeFailure := errors.New("fixture close failed") + for _, phase := range []string{"connect", "ping"} { + t.Run(phase, func(t *testing.T) { + owner := &fixtureConnector{closeErr: closeFailure} + if phase == "connect" { + owner.connect = func(context.Context) error { return failure } + } else { + owner.ping = func(context.Context) error { return failure } + } + pool, err := database.OpenConnector(nil, owner, database.DefaultPoolConfig()) + if pool != nil || !errors.Is(err, failure) || !errors.Is(err, closeFailure) { + t.Fatalf("lost failure or cleanup error: %v", err) + } + expectedConnections := int32(0) + if phase == "ping" { + expectedConnections = 1 + } + if owner.closedConnector.Load() != 1 || owner.closedConnections.Load() != expectedConnections { + t.Fatal("startup failure leaked resources") + } + }) + } +} + +func TestConnectorTimeoutBoundsConnectAndPing(t *testing.T) { + for _, phase := range []string{"connect", "ping"} { + t.Run(phase, func(t *testing.T) { + owner := &fixtureConnector{} + cfg := database.DefaultPoolConfig() + cfg.PingTimeout = 20 * time.Millisecond + wait := func(ctx context.Context) error { + if _, ok := ctx.Deadline(); !ok { + return errors.New("missing deadline") + } + <-ctx.Done() + return ctx.Err() + } + if phase == "connect" { + owner.connect = wait + } else { + owner.ping = wait + } + pool, err := database.OpenConnector(context.Background(), owner, cfg) + if pool != nil || !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("%s timeout: %v", phase, err) + } + if owner.closedConnector.Load() != 1 { + t.Fatal("timeout leaked connector") + } + }) + } +} + +func TestConnectorHealthBoundsPoolWaitAndRecovers(t *testing.T) { + cfg := database.DefaultPoolConfig() + cfg.MaxOpenConns = 1 + cfg.MaxIdleConns = 1 + cfg.PingTimeout = 20 * time.Millisecond + owner := &fixtureConnector{} + pool, err := database.OpenConnector(nil, owner, cfg) + if err != nil { + t.Fatal(err) + } + defer pool.Close() + conn, err := pool.Conn(context.Background()) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + if err = pool.Health(nil); !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("waiting for busy pool was not bounded: %v", err) + } + if err = conn.Close(); err != nil { + t.Fatal(err) + } + if err = pool.Health(nil); err != nil { + t.Fatal(err) + } +} + +func TestConnectorIdleZeroAndLifetime(t *testing.T) { + for _, mode := range []string{"noIdle", "expired"} { + t.Run(mode, func(t *testing.T) { + cfg := database.DefaultPoolConfig() + if mode == "noIdle" { + cfg.MaxIdleConns = 0 + } else { + cfg.ConnMaxLifetime = time.Nanosecond + } + owner := &fixtureConnector{} + pool, err := database.OpenConnector(nil, owner, cfg) + if err != nil { + t.Fatal(err) + } + defer pool.Close() + if err = pool.Health(nil); err != nil { + t.Fatal(err) + } + if owner.connects.Load() < 2 || owner.closedConnections.Load() == 0 { + t.Fatal("pool did not retire connections") + } + if mode == "noIdle" && pool.Stats().Idle != 0 { + t.Fatal("zero idle not honored") + } + if mode == "expired" && pool.Stats().MaxLifetimeClosed == 0 { + t.Fatal("lifetime not honored") + } + }) + } +} + +func TestConnectorTransactionsKeepCommitAndRollbackSemantics(t *testing.T) { + owner := &fixtureConnector{} + pool, err := database.OpenConnector(nil, owner, database.DefaultPoolConfig()) + if err != nil { + t.Fatal(err) + } + defer pool.Close() + ctx := context.Background() + if err = pool.InTx(ctx, nil, func(*sql.Tx) error { return nil }); err != nil { + t.Fatal(err) + } + failed := errors.New("business failed") + if err = pool.InTx(ctx, nil, func(*sql.Tx) error { return failed }); !errors.Is(err, failed) { + t.Fatal(err) + } + func() { + defer func() { + if recover() != failed { + t.Error("panic not preserved") + } + }() + _ = pool.InTx(ctx, nil, func(*sql.Tx) error { panic(failed) }) + }() + if owner.commits.Load() != 1 || owner.rollbacks.Load() != 2 { + t.Fatal("transaction semantics changed") + } +} diff --git a/database/database.go b/database/database.go index 8e1f080..1851d51 100644 --- a/database/database.go +++ b/database/database.go @@ -23,13 +23,14 @@ type Config struct { PingTimeout time.Duration `json:"pingTimeout"` } -func DefaultConfig() Config { - return Config{ - MaxOpenConns: 10, - MaxIdleConns: 5, - ConnMaxLifetime: 30 * time.Minute, - ConnMaxIdleTime: 5 * time.Minute, - PingTimeout: 5 * time.Second, +func DefaultConfig() Config { return DefaultPoolConfig().config() } + +// PoolConfig returns driver-independent connection pool settings. +func (c Config) PoolConfig() PoolConfig { + return PoolConfig{ + MaxOpenConns: c.MaxOpenConns, MaxIdleConns: c.MaxIdleConns, + ConnMaxLifetime: c.ConnMaxLifetime, ConnMaxIdleTime: c.ConnMaxIdleTime, + PingTimeout: c.PingTimeout, } } @@ -83,16 +84,7 @@ func (c Config) Validate() error { if strings.TrimSpace(c.DSN) == "" { return errors.New("database DSN is required") } - if c.MaxOpenConns < 0 || c.MaxIdleConns < 0 { - return errors.New("database pool limits cannot be negative") - } - if c.MaxIdleConns > c.MaxOpenConns && c.MaxOpenConns > 0 { - return errors.New("max idle connections cannot exceed max open connections") - } - if c.ConnMaxLifetime < 0 || c.ConnMaxIdleTime < 0 || c.PingTimeout <= 0 { - return errors.New("database timeouts are invalid") - } - return nil + return c.PoolConfig().Validate() } // DB owns a connection pool and exposes the embedded database/sql methods. @@ -112,23 +104,30 @@ func Open(ctx context.Context, cfg Config) (*DB, error) { if err != nil { return nil, fmt.Errorf("open database: %w", err) } - if cfg.MaxOpenConns > 0 { - db.SetMaxOpenConns(cfg.MaxOpenConns) - } - // database/sql uses zero to disable idle connections; do not leave its - // default pool size in place when the caller explicitly chooses zero. - db.SetMaxIdleConns(cfg.MaxIdleConns) + return initializePool(ctx, db, cfg) +} + +// initializePool is the shared ownership boundary for both creation paths. +// All configuration is validated before creating the sql.DB. +func initializePool(ctx context.Context, db *sql.DB, cfg Config) (*DB, error) { + db.SetMaxOpenConns(cfg.MaxOpenConns) + db.SetMaxIdleConns(cfg.MaxIdleConns) // Zero explicitly disables idle connections. db.SetConnMaxLifetime(cfg.ConnMaxLifetime) db.SetConnMaxIdleTime(cfg.ConnMaxIdleTime) checkCtx, cancel := context.WithTimeout(ctx, cfg.PingTimeout) defer cancel() if err := db.PingContext(checkCtx); err != nil { - _ = db.Close() - return nil, fmt.Errorf("ping database: %w", err) + pingErr := fmt.Errorf("ping database: %w", err) + if closeErr := db.Close(); closeErr != nil { + return nil, errors.Join(pingErr, fmt.Errorf("close database after failed ping: %w", closeErr)) + } + return nil, pingErr } return &DB{DB: db, config: cfg}, nil } +// Config returns the original DSN configuration. Pools opened with a Connector +// have empty Driver/DSN fields; their connector cannot be reconstructed from Config. func (db *DB) Config() Config { return db.config } func (db *DB) Health(ctx context.Context) error { diff --git a/database/mysqlConnector_test.go b/database/mysqlConnector_test.go new file mode 100644 index 0000000..b5ddcc3 --- /dev/null +++ b/database/mysqlConnector_test.go @@ -0,0 +1,180 @@ +package database_test + +import ( + "context" + "crypto/tls" + "crypto/x509" + "database/sql" + "errors" + "net" + "os" + "sync/atomic" + "testing" + "time" + + "git.yuntuo.hk/k2safe/go-omni/database" + "github.com/go-sql-driver/mysql" +) + +type connectorDriverLog struct{ count atomic.Int32 } + +func (l *connectorDriverLog) Print(...any) { l.count.Add(1) } + +func TestMySQLConnectorDriverLoggerIsolation(t *testing.T) { + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + stopped := make(chan struct{}) + go func() { + defer close(stopped) + for { + c, err := listener.Accept() + if err != nil { + return + } + c.Close() + } + }() + defer func() { listener.Close(); <-stopped }() + logs := [2]connectorDriverLog{} + for i := range logs { + cfg := mysql.NewConfig() + cfg.Net = "tcp" + cfg.Addr = listener.Addr().String() + cfg.Timeout = time.Second + cfg.ReadTimeout = time.Second + cfg.Logger = &logs[i] + connector, err := mysql.NewConnector(cfg) + if err != nil { + t.Fatal(err) + } + before := [2]int32{logs[0].count.Load(), logs[1].count.Load()} + db, err := database.OpenConnector(context.Background(), connector, database.DefaultPoolConfig()) + if err == nil { + db.Close() + t.Fatal("handshake to closed socket succeeded") + } + if logs[i].count.Load() <= before[i] || logs[1-i].count.Load() != before[1-i] { + t.Fatal("driver logger did not stay on its connector") + } + } +} + +func TestMySQLConnectorTLSAndPoolIsolation(t *testing.T) { + if os.Getenv("GO_OMNI_CONNECTOR_MYSQL_TEST") != "1" { + t.Skip("use scripts/verifyConnector.py for isolated MySQL TLS verification") + } + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second) + defer cancel() + ca, err := os.ReadFile(os.Getenv("GO_OMNI_CONNECTOR_CA_FILE")) + if err != nil { + t.Fatal(err) + } + roots := x509.NewCertPool() + if !roots.AppendCertsFromPEM(ca) { + t.Fatal("invalid test CA") + } + cfg := mysql.NewConfig() + cfg.Net = "tcp" + cfg.Addr = os.Getenv("GO_OMNI_CONNECTOR_ADDR") + cfg.User = os.Getenv("GO_OMNI_CONNECTOR_USER") + cfg.Passwd = os.Getenv("GO_OMNI_CONNECTOR_PASSWORD") + cfg.DBName = os.Getenv("GO_OMNI_CONNECTOR_DATABASE") + cfg.Timeout = time.Second + cfg.ReadTimeout = 2 * time.Second + cfg.WriteTimeout = 2 * time.Second + cfg.TLS = &tls.Config{MinVersion: tls.VersionTLS12, ServerName: "localhost", RootCAs: roots} + pool := database.DefaultPoolConfig() + pool.MaxOpenConns = 2 + pool.MaxIdleConns = 2 + pool.PingTimeout = 2 * time.Second + open := func(cfg *mysql.Config) (*database.DB, error) { + connector, err := mysql.NewConnector(cfg) + if err != nil { + return nil, err + } + return database.OpenConnector(ctx, connector, pool) + } + cfg.Params = map[string]string{"time_zone": "'+00:00'"} + first, err := open(cfg) + if err != nil { + t.Fatal(err) + } + defer first.Close() + other := cfg.Clone() + other.Params["time_zone"] = "'+08:00'" + second, err := open(other) + if err != nil { + t.Fatal(err) + } + defer second.Close() + for i, db := range []*database.DB{first, second} { + var name, cipher, zone string + if err := db.QueryRowContext(ctx, "SHOW SESSION STATUS LIKE 'Ssl_cipher'").Scan(&name, &cipher); err != nil { + t.Fatal(err) + } + if cipher == "" { + t.Fatal("connection is not encrypted") + } + if err := db.QueryRowContext(ctx, "SELECT @@session.time_zone").Scan(&zone); err != nil { + t.Fatal(err) + } + if zone != []string{"+00:00", "+08:00"}[i] { + t.Fatalf("session config leaked: %s", zone) + } + if db.Stats().MaxOpenConnections != 2 { + t.Fatal("pool configuration ignored") + } + } + for _, bad := range []string{"untrustedCA", "wrongHostname"} { + t.Run(bad, func(t *testing.T) { + invalid := cfg.Clone() + if bad == "untrustedCA" { + invalid.TLS.RootCAs = x509.NewCertPool() + } else { + invalid.TLS.ServerName = "wrong.invalid" + } + db, err := open(invalid) + if err == nil { + db.Close() + t.Fatal("invalid server certificate accepted") + } + var verification *tls.CertificateVerificationError + if !errors.As(err, &verification) { + t.Fatalf("not a TLS verification failure: %v", err) + } + if err := first.Health(ctx); err != nil { + t.Fatalf("failed connector affected existing pool: %v", err) + } + }) + } + if _, err := first.ExecContext(ctx, "CREATE TABLE connector_check (id INT PRIMARY KEY)"); err != nil { + t.Fatal(err) + } + if err := first.InTx(ctx, nil, func(tx *sql.Tx) error { + _, err := tx.ExecContext(ctx, "INSERT INTO connector_check VALUES (1)") + return err + }); err != nil { + t.Fatal(err) + } + rollbackErr := errors.New("rollback requested") + if err := first.InTx(ctx, nil, func(tx *sql.Tx) error { + if _, err := tx.ExecContext(ctx, "INSERT INTO connector_check VALUES (2)"); err != nil { + return err + } + return rollbackErr + }); !errors.Is(err, rollbackErr) { + t.Fatal(err) + } + var count int + if err := second.QueryRowContext(ctx, "SELECT COUNT(*) FROM connector_check").Scan(&count); err != nil || count != 1 { + t.Fatalf("transaction count=%d err=%v", count, err) + } + if err := first.Close(); err != nil { + t.Fatal(err) + } + if err := second.Health(ctx); err != nil { + t.Fatal("closing first pool affected second:", err) + } +} diff --git a/omni.go b/omni.go index 2865f83..20a9289 100644 --- a/omni.go +++ b/omni.go @@ -26,6 +26,7 @@ type HTTPRouterGroup = httpx.RouterGroup type APIError = apierrors.APIError type Config = config.Config type TLSConfig = config.TLSConfig +type HTTPConfig = config.HTTPConfig type Option = config.Option type Response = response.Response type LogConfig = logx.Config @@ -62,6 +63,8 @@ func WithConfig(settings Config) Option { return config.WithConfig(settings) } func WithTransport(name string) Option { return config.WithTransport(name) } +func WithHTTPConfig(settings HTTPConfig) Option { return config.WithHTTPConfig(settings) } + func WithTLS(certFile, keyFile string) Option { return config.WithTLS(certFile, keyFile) } func WithLogConfig(settings LogConfig) Option { return config.WithLogConfig(settings) } diff --git a/performance/connectorAcceptance.md b/performance/connectorAcceptance.md new file mode 100644 index 0000000..8ad690c --- /dev/null +++ b/performance/connectorAcceptance.md @@ -0,0 +1,24 @@ +# v0.9.1-preview Connector 与 HTTP 配置验收 + +## 范围 + +新增标准 `driver.Connector` 注入和 HTTP 服务器读写配置；没有修改 eTax/ePay 业务代码。数据库驱动/TLS/Logger/时区/版本校验仍归项目。网络测试只监听本地回环地址，MySQL 使用独立本地容器（1 CPU / 768 MiB / 128 PID），没有使用业务服务器压测。 + +## 全量回归 + +- `GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=1 -count=1 -timeout=300s ./...`：40 个有测试的包全部通过，包含真实脚手架/契约生成的集成消费者。 +- `go vet ./...`、`go build ./...`、gofmt、`git diff --check` 通过。 + +## 专项验收 + +- `go test -race -count=3 ./database ./app ./config` 通过。 +- 两个 Connector 并发读写保持独立；启动 Connect/Ping 与查询 context 保留，后台补连接遵循 database/sql 行为；最大连接数、禁用 idle、连接生命周期、健康检查等待上限及恢复验证通过。 +- nil/带类型的 nil、无效参数、启动前取消不取得 Connector 所有权；Connect/Ping 失败与超时关闭池，清理错误和原错误可同时检查；成功关闭幂等，事务提交/错误/panic 回滚通过。 +- 标准库服务器字段、ErrorLog 透传、无效配置启动前返回并清理资源、JSON/YAML/环境变量覆盖通过；不支持的独立 http2 选项明确拒绝。 +- nethttp/native 的实际 TCP 慢请求头、慢 body、Keep-Alive 空闲关闭、Flush 后客户端停止读取触发写超时、超限头 431 通过。测试不假定两种网络层有完全相同的超时响应格式。 +- `scripts/verifyConnector.py --local` 的独立消费者通过：真实 MySQL 8.4 TLS、错误 CA/主机名拒绝、不同 session time_zone、每个 Connector 的驱动 Logger、两池独立关闭、事务提交/回滚。所有专项测试无跳过，详情见 [本地记录](connectorLocal.json)。 +- MySQL 镜像固定记录为 `sha256:1afd25fe334a1be64a3866495dd43a6bdba3c55dd1d6190ea12933163fe2843c`。临时容器、卷、证书、凭据及消费者目录已清理。 + +环境：`go version go1.27.1 darwin/arm64`。记录时间：2026-09-23T18:17:16.897323+00:00。 + +独立 `TransportHTTP2` 的逐请求读写时限未实现，本次没有进行 Linux soak、代理长测或完整业务迁移验收。 diff --git a/performance/connectorLocal.json b/performance/connectorLocal.json new file mode 100644 index 0000000..84722f2 --- /dev/null +++ b/performance/connectorLocal.json @@ -0,0 +1,70 @@ +{ + "passed": true, + "frameworkVersion": "v0.9.1-preview", + "localSource": true, + "frameworkSum": null, + "freshModuleCache": false, + "noLocalReplace": false, + "mysqlImage": "sha256:1afd25fe334a1be64a3866495dd43a6bdba3c55dd1d6190ea12933163fe2843c", + "mysqlLimits": { + "cpus": 1, + "memoryMiB": 768, + "pids": 128 + }, + "durationSeconds": 9.61, + "tests": [ + "TestHTTPFileAndEnvironment/server.json", + "TestHTTPFileAndEnvironment/server.yaml", + "TestHTTPFileAndEnvironment", + "TestHTTPInvalidAndUnsupportedSettings/READ_HEADER_TIMEOUTinvalid", + "TestHTTPInvalidAndUnsupportedSettings/READ_HEADER_TIMEOUT-1", + "TestHTTPInvalidAndUnsupportedSettings/READ_TIMEOUTinvalid", + "TestHTTPInvalidAndUnsupportedSettings/READ_TIMEOUT-1", + "TestHTTPInvalidAndUnsupportedSettings/WRITE_TIMEOUTinvalid", + "TestHTTPInvalidAndUnsupportedSettings/WRITE_TIMEOUT-1", + "TestHTTPInvalidAndUnsupportedSettings/IDLE_TIMEOUTinvalid", + "TestHTTPInvalidAndUnsupportedSettings/IDLE_TIMEOUT-1", + "TestHTTPInvalidAndUnsupportedSettings/MAX_HEADER_BYTESinvalid", + "TestHTTPInvalidAndUnsupportedSettings/MAX_HEADER_BYTES-1", + "TestHTTPInvalidAndUnsupportedSettings", + "TestHTTPConfiguredStandardServer", + "TestHTTPInvalidStartupReleasesResources", + "TestHTTPConfiguredNetworkLimits/nethttp/slowHeaders", + "TestHTTPConfiguredNetworkLimits/nethttp/slowBody", + "TestHTTPConfiguredNetworkLimits/nethttp/idle", + "TestHTTPConfiguredNetworkLimits/nethttp/writeBackpressure", + "TestConnectorIndependentPoolsAndContext", + "TestConnectorRejectsInvalidInputWithoutTakingOwnership", + "TestConnectorStartupFailuresReleaseResources/connect", + "TestConnectorStartupFailuresReleaseResources/ping", + "TestConnectorStartupFailuresReleaseResources", + "TestConnectorTimeoutBoundsConnectAndPing/connect", + "TestConnectorTimeoutBoundsConnectAndPing/ping", + "TestConnectorTimeoutBoundsConnectAndPing", + "TestConnectorHealthBoundsPoolWaitAndRecovers", + "TestConnectorIdleZeroAndLifetime/noIdle", + "TestConnectorIdleZeroAndLifetime/expired", + "TestConnectorIdleZeroAndLifetime", + "TestConnectorTransactionsKeepCommitAndRollbackSemantics", + "TestMySQLConnectorDriverLoggerIsolation", + "TestMySQLConnectorTLSAndPoolIsolation/untrustedCA", + "TestMySQLConnectorTLSAndPoolIsolation/wrongHostname", + "TestMySQLConnectorTLSAndPoolIsolation", + "TestHTTPConfiguredNetworkLimits/nethttp/headerLimit", + "TestHTTPConfiguredNetworkLimits/nethttp", + "TestHTTPConfiguredNetworkLimits/native/slowHeaders", + "TestHTTPConfiguredNetworkLimits/native/slowBody", + "TestHTTPConfiguredNetworkLimits/native/idle", + "TestHTTPConfiguredNetworkLimits/native/writeBackpressure", + "TestHTTPConfiguredNetworkLimits/native/headerLimit", + "TestHTTPConfiguredNetworkLimits/native", + "TestHTTPConfiguredNetworkLimits" + ], + "sourceHashes": { + "database/connector_test.go": "0663dfc9794ba24130225fa3b6418a6d5f75d57d0f01686151f70c7ea3f592c0", + "database/mysqlConnector_test.go": "e1cdd11ef76b09f1332ba92bfdde5b1917d84c85cc42fac48f65a8b23f5e9720", + "app/httpConfig_test.go": "b060dc0b76fb8e5feebae7e3b12cb7bca845693fcc097e2a0496a5ba937c837a", + "config/http_test.go": "0cae3dcf213369cfaff5487bf630c2bc3ec097739ca1d40794124e69c3880db8", + "scripts/verifyConnector.py": "a0ba3fa29a4d1efcbed5dad82c09afb97d8ddecd904f20fee1e3aa74ec0f9e06" + } +} diff --git a/scripts/verifyConnector.py b/scripts/verifyConnector.py new file mode 100644 index 0000000..e827700 --- /dev/null +++ b/scripts/verifyConnector.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +"""Verify Connector TLS/isolation and HTTP limits from a local or remote module. + +Uses a disposable local MySQL container (1 CPU, 768 MiB), private test credentials, +and generated test certificates. Never connects to a project database. +""" +import argparse +import hashlib +import json +import os +from pathlib import Path +import re +import secrets +import shutil +import subprocess +import tempfile +import time +import uuid + +ROOT = Path(__file__).resolve().parents[1] +MODULE = "git.yuntuo.hk/k2safe/go-omni" +SOURCES = ("database/connector_test.go", "database/mysqlConnector_test.go", + "app/httpConfig_test.go", "config/http_test.go") + + +def run(args, cwd, env, timeout=300): + result = subprocess.run([str(a) for a in args], cwd=cwd, env=env, + text=True, capture_output=True, timeout=timeout) + output = re.sub(r"https://[^/@\s]+:[^/@\s]+@", "https://", result.stdout + result.stderr) + # Never print generated database credentials, including from a failed command. + for key in ("MYSQL_ROOT_PASSWORD", "MYSQL_PASSWORD", "GO_OMNI_CONNECTOR_PASSWORD"): + if env.get(key): + output = output.replace(env[key], "[redacted]") + if result.returncode: + raise RuntimeError(f"{args[0]} failed ({result.returncode}): {output}") + return output + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + source = parser.add_mutually_exclusive_group(required=True) + source.add_argument("--local", action="store_true") + source.add_argument("--version") + parser.add_argument("--report", type=Path, required=True) + parser.add_argument("--image", default="mysql:8.4") + args = parser.parse_args() + if args.version and not re.fullmatch(r"v\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?", args.version): + parser.error("a fixed release version is required") + env = {k: v for k, v in os.environ.items() if not k.startswith("GO_OMNI_")} + env.update(GOWORK="off", GOFLAGS="") + for key in ("GOPRIVATE", "GONOSUMDB"): + env[key] = ",".join(filter(None, [env.get(key), "git.yuntuo.hk"])) + began = time.monotonic() + with tempfile.TemporaryDirectory(prefix="goOmniConnector-") as directory: + temp = Path(directory) + endpoint = env.get("DOCKER_HOST") or run(["docker", "context", "inspect", "--format", "{{.Endpoints.docker.Host}}"], temp, env).strip() + if not endpoint.startswith(("unix://", "npipe://")): + raise RuntimeError("requires local Docker; remote servers are not used") + certs = temp / "certs" + certs.mkdir(mode=0o755) + run(["openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", certs / "ca.key", "-out", certs / "ca.pem", "-days", "2", "-subj", "/CN=GoOmni Test CA"], temp, env) + run(["openssl", "req", "-newkey", "rsa:2048", "-nodes", "-keyout", certs / "server.key", "-out", temp / "server.csr", "-subj", "/CN=localhost"], temp, env) + (temp / "extensions.cnf").write_text("subjectAltName=DNS:localhost,IP:127.0.0.1\nbasicConstraints=CA:FALSE\nextendedKeyUsage=serverAuth\n") + run(["openssl", "x509", "-req", "-in", temp / "server.csr", "-CA", certs / "ca.pem", "-CAkey", certs / "ca.key", "-CAcreateserial", "-out", certs / "server.pem", "-days", "2", "-extfile", temp / "extensions.cnf"], temp, env) + (certs / "ca.key").unlink() + for path in certs.iterdir(): + path.chmod(0o644) # Disposable fixtures, readable by the container's mysql uid. + project = temp / "consumer" + project.mkdir() + version = args.version or "v0.9.1-preview" + (project / "go.mod").write_text(f"module example.com/connectorAcceptance\n\ngo 1.22.0\n\nrequire {MODULE} {version}\n") + if args.local: + run(["go", "mod", "edit", f"-replace={MODULE}={ROOT}"], project, env) + else: + env["GOMODCACHE"] = str(temp / "modcache") + for name in SOURCES: + target = project / name + target.parent.mkdir(exist_ok=True) + shutil.copyfile(ROOT / name, target) + print("Resolving consumer dependency", flush=True) + run(["go", "mod", "tidy"], project, env) + module = json.loads(run(["go", "list", "-m", "-json", MODULE], project, env)) + if not args.local: + assert module["Version"] == args.version and module.get("Sum") and not module.get("Replace"), module + env.update(MYSQL_ROOT_PASSWORD=secrets.token_hex(24), MYSQL_PASSWORD=secrets.token_hex(24)) + name = "go-omni-connector-" + uuid.uuid4().hex[:12] + created = False + try: + print("Starting isolated MySQL TLS fixture (1 CPU, 768 MiB)", flush=True) + run(["docker", "create", "--name", name, "--label", "go-omni.test=connector", "--cpus", "1", "--memory", "768m", "--pids-limit", "128", "--publish", "127.0.0.1::3306", "--mount", f"type=bind,src={certs},dst=/certs,readonly", "--env", "MYSQL_ROOT_PASSWORD", "--env", "MYSQL_PASSWORD", "--env", "MYSQL_DATABASE=omni_test", "--env", "MYSQL_USER=omni_test", args.image, "--ssl-ca=/certs/ca.pem", "--ssl-cert=/certs/server.pem", "--ssl-key=/certs/server.key", "--require-secure-transport=ON", "--innodb-buffer-pool-size=64M", "--max-connections=20"], temp, env) + created = True + run(["docker", "start", name], temp, env) + ready = False + for _ in range(60): + try: + check_env = dict(env, MYSQL_PWD=env["MYSQL_PASSWORD"]) + result = run(["docker", "exec", "--env", "MYSQL_PWD", name, "mysql", "--protocol=tcp", "-h127.0.0.1", "-uomni_test", "--batch", "--skip-column-names", "-e", "SELECT 1"], temp, check_env, timeout=5) + if result.strip() == "1": + ready = True + break + except (RuntimeError, subprocess.TimeoutExpired): + pass + time.sleep(1) + if not ready: + raise RuntimeError("isolated MySQL fixture did not become ready") + addr = run(["docker", "port", name, "3306/tcp"], temp, env).strip() + assert addr.startswith("127.0.0.1:"), addr + env.update(GO_OMNI_CONNECTOR_MYSQL_TEST="1", GO_OMNI_CONNECTOR_ADDR=addr, + GO_OMNI_CONNECTOR_USER="omni_test", GO_OMNI_CONNECTOR_PASSWORD=env["MYSQL_PASSWORD"], + GO_OMNI_CONNECTOR_DATABASE="omni_test", GO_OMNI_CONNECTOR_CA_FILE=str(certs / "ca.pem")) + print("Testing TLS trust, per-connector options, driver logging, lifecycle and HTTP limits", flush=True) + output = run(["go", "test", "-race", "-count=1", "-json", "-timeout=60s", "./..."], project, env) + events = [json.loads(line) for line in output.splitlines() if line.startswith("{")] + passed = [e["Test"] for e in events if e.get("Action") == "pass" and "Test" in e] + assert "TestMySQLConnectorTLSAndPoolIsolation" in passed + assert not any(e.get("Action") in ("fail", "skip") for e in events), output + image = run(["docker", "inspect", "--format", "{{.Image}}", name], temp, env).strip() + finally: + if created: + run(["docker", "rm", "--force", "--volumes", name], temp, env) + report = {"passed": True, "frameworkVersion": version, "localSource": args.local, + "frameworkSum": module.get("Sum"), "freshModuleCache": not args.local, + "noLocalReplace": not bool(module.get("Replace")), "mysqlImage": image, + "mysqlLimits": {"cpus": 1, "memoryMiB": 768, "pids": 128}, + "durationSeconds": round(time.monotonic() - began, 2), "tests": passed, + "sourceHashes": {name: hashlib.sha256((ROOT / name).read_bytes()).hexdigest() for name in (*SOURCES, "scripts/verifyConnector.py")}} + args.report.parent.mkdir(parents=True, exist_ok=True) + args.report.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n") + print("PASS; isolated container, credentials, certificates and consumer removed", flush=True) + + +if __name__ == "__main__": + main() diff --git a/transport/README.md b/transport/README.md index a14a590..9dbc27f 100644 --- a/transport/README.md +++ b/transport/README.md @@ -14,7 +14,7 @@ registerRoutes(application) // 与 omni.New() 复用同一份业务注册代码 `transport/nethttp` 是当前内置的标准库路由实现。它提供一套可移植的路径、方法和错误回退规则；自研网络层可以直接实现同一个 `httpx.Router` 契约。 -`transport/http2` 是独立的 opt-in 实验 Runner，负责连接接入、TLS/ALPN、连接上限和全局请求背压，底层帧解析与 stream 流控暂由 `golang.org/x/net/http2` 提供。它不替换 `transport/native`，也不改变默认启动器；只有通过 `App.RunWith(ctx, http2.New(nil), address)` 显式启用才会运行。 +`transport/http2` 是独立的 opt-in 实验 Runner，负责连接接入、TLS/ALPN、连接上限和全局请求背压，底层帧解析与 stream 流控暂由 `golang.org/x/net/http2` 提供。它不替换 `transport/native`，也不改变默认启动器；通过 `WithTransport(TransportHTTP2)` 或 `App.RunWith(ctx, http2.New(nil), address)` 显式启用后运行。 `transport/native` 是第一版自研 HTTP/1.1 服务端：它自己管理 TCP listener、连接、Keep-Alive、请求解析、响应写回和优雅关闭，再把请求交给现有 `http.Handler`。请求体支持 `Content-Length` 和 chunked，并统一受 `MaxBodyBytes` 限制；其他传输编码会被拒绝。它仍使用标准 `http.Request` / `http.ResponseWriter` 作为业务桥接，因此不会影响现有 Handler / Service。 @@ -119,3 +119,37 @@ python3 scripts/verifyTransports.py --version v0.6.0-preview 第一条验证标准库适配器复用完整 Handler / Service / Repository。第二条在源码目录外创建无 replace 的消费模块，从远程版本导入同一套验收包；自研适配器也应使用同一个公共测试包。 当前替换边界分成两层：`transport/nethttp` 提供标准库路由，`transport/native` 进一步接管 TCP 监听、连接管理和 HTTP/1.1 协议 I/O。`App.Run` / `App.RunContext` 仍使用标准库 `net/http.Server`，需要显式调用 `App.RunWith(ctx, native.New(nil), address)` 才会启用 native 预览实现。native 当前通过标准 `http.Request` / `http.ResponseWriter` 桥接到业务层，后续可在不改 Handler / Service 的前提下逐步替换解析器和响应写回。 + +## HTTP 配置（v0.9.1-preview） + +`omni.WithHTTPConfig` / `config.Config.HTTP` 提供服务端配置入口，作用于 `Run`、`RunContext`、`ServeContext`；也作用于 `NewServer` 创建的标准库服务器。与客户端 `httpc` 的出站请求超时无关。 + +```go +application := omni.New(omni.WithHTTPConfig(omni.HTTPConfig{ + ReadHeaderTimeout: 5 * time.Second, + ReadTimeout: 30 * time.Second, + WriteTimeout: 30 * time.Second, + IdleTimeout: 60 * time.Second, + MaxHeaderBytes: 1 << 20, + // ErrorLog: existingLogger, // *log.Logger，仅 nethttp；不通过 YAML/JSON 序列化 +})) +// 注册业务路由后： +return application.RunContext(ctx, ":8080") +``` + +| 字段 | nethttp | native | 独立实验 http2 | +| --- | --- | --- | --- | +| `ReadHeaderTimeout` | 标准 http.Server 语义 | 请求头读取时限 | 正值拒绝启动 | +| `ReadTimeout` | 整个请求读取时限，包括头与 body | 头解析完成后读取 body 的时限 | 正值拒绝启动 | +| `WriteTimeout` | 标准 http.Server 写截止时间 | 每次底层响应写入/Flush 的截止时间 | 正值拒绝启动 | +| `IdleTimeout` | Keep-Alive 等待下一请求 | 复用连接等待下一请求（含头读取）；首个请求读头也取它与读头时限的较小值 | HTTP/2 空闲连接超时 | +| `MaxHeaderBytes` | 标准 http.Server 头限制语义 | native 请求头字节限制 | 解压后请求头限制 | +| `ErrorLog` | 标准库内部错误日志，不是访问日志 | 非 nil 拒绝启动 | 非 nil 拒绝启动 | + +参数为正时覆盖适配器默认值；**零值保留默认行为，不统一表示禁用**。nethttp 未配置时遵循 Go 标准服务器零值，包括读头/空闲时限回退到 ReadTimeout 的规则；native 仍默认读头 5s、读 body 30s、写 30s、空闲 60s、头 1 MiB。负数在启动前报错；HTTP/2 不支持的字段也会明确报错，不会悄悄忽略。默认 nethttp 的 TLS/HTTP2 行为仍由标准库管理，表格最后一列专指 `TransportHTTP2`。 + +I/O 超时不会强制终止 CPU 计算或数据库调用；业务和下游调用继续通过 `context` 控制时限。native 需要流式输出时显式 `Flush()`，否则保持原有缓冲响应方式。不要依赖不同网络层在慢请求时返回完全相同的 HTTP 错误体；有些超时会直接断开连接。 + +YAML/JSON 字段与上述字段对应的 lowerCamelCase 一致，位于 `config.Config` 的 `http` 对象。脚手架将框架配置包在 `server` 下，因此生成项目填写 `server.http`；直接用 `config.LoadFile` 加载框架配置时填写根级 `http`。环境变量为 `GO_OMNI_HTTP_READ_HEADER_TIMEOUT`、`GO_OMNI_HTTP_READ_TIMEOUT`、`GO_OMNI_HTTP_WRITE_TIMEOUT`、`GO_OMNI_HTTP_IDLE_TIMEOUT`（如 `30s`），以及 `GO_OMNI_HTTP_MAX_HEADER_BYTES`（整数）。`omni.New` 不自动加载环境；先 `config.LoadFile` / `config.ApplyEnv`，再 `WithConfig`。Go-only 的 `ErrorLog` 在文件加载之后设置。 + +`NewServer` 始终返回普通 `*http.Server`，应用 HTTP 字段但不负责加载 TLS、监听、退出或资源清理；非法 HTTP 值会 panic。`RunContext` / `ServeContext` 的非法配置则作为 error 返回并关闭 App 资源。手动修改返回服务器的字段仍然有效。项目自己的 Runner 通过 `RunWith` 接入时，配置由该 Runner 持有，这组 HTTP 配置不会覆盖它。