feat: support database connectors and HTTP server configuration

This commit is contained in:
2026-09-24 02:18:18 +08:00
parent 6200864a64
commit 20e84e5ff8
22 changed files with 1403 additions and 63 deletions

View File

@@ -1,5 +1,14 @@
# Changelog
## v0.9.1-preview
- 新增标准 `database.OpenConnector` 和独立 `PoolConfig`;与 DSN 入口共享池参数、限时启动 Ping、失败清理、Health、事务,不注册全局驱动配置。
- 新增 `omni.WithHTTPConfig` / `Config.HTTP`:读头、读请求、写响应、空闲超时和头大小,贯通代码、JSON/YAML、环境变量、Run/Serve 与 NewServer;nethttp 支持 Go-only ErrorLog。
- 保持已有默认行为;负值和实验 http2 不支持的超时项在监听前报错;记录各网络层的具体语义。
- 增加真实 MySQL TLS/驱动日志隔离、连接池归属/取消/事务、nethttp/native 慢连接与写背压验收,以及无本地 replace 的远程消费者脚本。
- 兼容 v0.9.0-preview 生成代码;不修改业务项目的数据库策略或生成代码最低版本。
## v0.9.0-preview
- 标准 HTTP Handler/Middleware 桥接:跟踪 Raw 状态与提交,保留取消、流式能力和中间件跨请求状态;已提交响应不追加 JSON。

View File

@@ -2,7 +2,7 @@
GoOmni 是面向多个业务项目复用的 Go 后端框架:用 `.api` 定义接口、请求和返回类型,生成项目目录、HTTP 接入代码及 OpenAPI,业务代码负责 Service 和 Repository。也可以在已有项目中直接使用 `omni.New()`,或单独引入日志、存储等模块。
**从这份 README 开始即可接入。** 当前版本为 **`v0.9.0-preview`**,Go **1.22+**;CLI 和项目依赖使用同一固定版本。框架不依赖 Gin,默认使用标准库网络层,提供可切换的 native HTTP/1.1 和实验性 HTTP/2 适配器。
**从这份 README 开始即可接入。** 当前版本为 **`v0.9.1-preview`**,Go **1.22+**;CLI 和项目依赖使用同一固定版本。框架不依赖 Gin,默认使用标准库网络层,提供可切换的 native HTTP/1.1 和实验性 HTTP/2 适配器。
响应字段、业务码、登录方式、权限规则和数据库表结构由接入项目决定。文中的 `code/msg/data`、`200/9999` 都是示例配置,可以替换。
@@ -26,7 +26,7 @@ module 地址是 `git.yuntuo.hk/k2safe/go-omni`。先确保本机 Go 和 Git 可
```sh
export GOPRIVATE=git.yuntuo.hk
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.1-preview
export PATH="$(go env GOPATH)/bin:$PATH"
go-omni --help
```
@@ -38,7 +38,7 @@ go-omni --help
在准备存放业务项目的目录执行,`./hello` 必须是新目录或空目录;把 `example.com/hello` 换成自己的 Go module 名称。
```sh
go-omni api new --framework-version v0.9.0-preview \
go-omni api new --framework-version v0.9.1-preview \
--module example.com/hello ./hello
cd hello
go mod tidy
@@ -126,7 +126,7 @@ service catalog-api {
```sh
go-omni api validate --api catalog.api
go-omni new --api catalog.api --framework-version v0.9.0-preview \
go-omni new --api catalog.api --framework-version v0.9.1-preview \
--module example.com/catalog ./catalog
cd catalog
go mod tidy
@@ -167,21 +167,21 @@ curl -i http://127.0.0.1:8080/api/v1/items/missing
需要 MySQL、文件存储或文件日志时,**首次生成**可以使用:
```sh
go-omni new --api catalog.api --framework-version v0.9.0-preview \
go-omni new --api catalog.api --framework-version v0.9.1-preview \
--module example.com/catalogFull --database mysql --storage local \
--log-output file --log-format json ./catalogFull
```
该命令在 `catalog.api` 所在目录执行,目标仍需是新目录。数据库迁移、表结构和 Repository 由项目维护,API 模板不自动建表;MySQL DSN 通过 `GO_OMNI_DB_DSN` 提供。存储还可选 `s3`、`r2`、`oss`,按生成项目的配置说明提供桶、端点和凭据。完整装配例子见 [userService](examples/userService/README.md)。
不采用 `.api` 时,也可以用 `go-omni new --framework-version v0.9.0-preview --module example.com/billing ./billing` 创建手写接口的 CRUD 示例项目;两种生成方式选择一种即可。
不采用 `.api` 时,也可以用 `go-omni new --framework-version v0.9.1-preview --module example.com/billing ./billing` 创建手写接口的 CRUD 示例项目;两种生成方式选择一种即可。
## 已有项目:按模块接入
在业务项目的 **go.mod 所在目录**执行:
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.9.0-preview
go get git.yuntuo.hk/k2safe/go-omni@v0.9.1-preview
```
已有项目可直接复用现有目录、Service、Repository 和数据库连接,先选择一个接口接入。也可以只用 `logx`、`storage` 等包,暂时保留原来的 HTTP 框架。
@@ -313,6 +313,8 @@ auth UserAuth (
依赖在启动入口或项目自己的 `internal/servicecontext` 中创建,通过构造函数传给 Service/Repository,避免每个请求都新建连接池。只注册自己拥有的资源,例如 `app.OnShutdown(db.Close)`;关闭按注册的逆序执行。
已有项目使用 `driver.Connector` 时,直接 `database.OpenConnector(ctx, connector, database.DefaultPoolConfig())`。MySQL 等驱动的独立 TLS、Logger、时区与拨号参数由项目构造 Connector;框架负责连接池、启动 Ping、健康检查及事务。无需转成 DSN,也无需注册全局 TLS/Logger。具体示例和资源归属见 [Connector 接入](database/README.md#connector-接入v091-preview)。
| 能力 | 包与接入说明 |
| --- | --- |
| 数据库连接池、事务、显式迁移 | `database`,见 [数据库](database/README.md);表结构与驱动由项目选择 |
@@ -332,6 +334,9 @@ API 项目的数据库迁移需要单独维护并显式执行;启动不会自
| `GO_OMNI_HTTP_ADDR` | 监听地址;生成项目默认 `127.0.0.1:8080` |
| `GO_OMNI_TRANSPORT` | `nethttp`(默认)、`native`、`http2` |
| `GO_OMNI_TLS_CERT_FILE` / `GO_OMNI_TLS_KEY_FILE` | 成对设置 TLS 证书和私钥路径 |
| `GO_OMNI_HTTP_READ_HEADER_TIMEOUT` / `GO_OMNI_HTTP_READ_TIMEOUT` | 请求头 / 请求读取超时,如 `5s` / `30s` |
| `GO_OMNI_HTTP_WRITE_TIMEOUT` / `GO_OMNI_HTTP_IDLE_TIMEOUT` | 响应写入 / Keep-Alive 空闲超时 |
| `GO_OMNI_HTTP_MAX_HEADER_BYTES` | 请求头大小限制,字节数 |
| `GO_OMNI_SHUTDOWN_TIMEOUT` | 优雅退出等待时间,默认 `10s` |
| `GO_OMNI_ENABLE_LOGGER` / `GO_OMNI_LOG_LEVEL` | 访问日志开关、日志级别 |
| `GO_OMNI_TRUSTED_PROXIES` | 可解析转发地址头的可信代理 IP/CIDR,逗号分隔 |
@@ -345,6 +350,20 @@ GO_OMNI_TRANSPORT=native go run ./cmd/server
手写入口可用 `omni.WithTransport(omni.TransportNative)`;TLS 配置为 `omni.WithTLS(certFile, keyFile)`。native 提供 HTTP/1.1、Keep-Alive、TLS、超时、连接/请求并发上限和对象复用;HTTP/2 是单独的实验适配器,必须配置 TLS。默认 nethttp 的 TLS 支持 HTTP/1.1 与 HTTP/2。
读写超时也可以在生成项目的 `etc/config.yaml` 中声明(放在 `server.http` 下):
```yaml
server:
http:
readHeaderTimeout: 5s
readTimeout: 30s
writeTimeout: 30s
idleTimeout: 60s
maxHeaderBytes: 1048576
```
代码配置使用 `omni.WithHTTPConfig(omni.HTTPConfig{ReadTimeout: 30*time.Second, WriteTimeout: 30*time.Second})`。`Run`、`RunContext`、`ServeContext` 会应用它;`NewServer` 会配置返回的标准 `http.Server`,包含可选的 Go 字段 `ErrorLog`。**正值覆盖、零值保留所选网络层的原有默认行为、负值报错**;不配置时升级行为不变。超时含义与支持范围见 [HTTP 配置](transport/README.md#http-配置v091-preview),不要把网络 I/O 超时当成业务执行时限。
自研网络层通过 `RunWith(ctx, runner)` 接入监听与连接生命周期,通过 `NewWithRouter(router)` 替换路由;业务 Handler/Service 不需要导入具体适配器。细节与验收契约见 [网络适配层](transport/README.md)。
`Run` 处理 SIGINT/SIGTERM;`RunContext` 接收调用方控制的退出信号。服务先停止接收新请求,等待活跃请求完成,再清理资源;超时会取消请求并关闭连接,业务代码仍需响应 context 取消。
@@ -364,7 +383,7 @@ go test ./...
`Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理,手动修改会报告冲突。Service、Repository、中间件实现、main、配置和依赖装配属于项目,重复生成保留手写内容;契约改变后仍需自行调整业务签名并通过编译。将 `.api`、生成文件和 `.goOmni/apiManifest.json` 一起提交,CI 可运行 `generate --check` 检查是否同步。
升级时先安装目标版本的 CLI,再在业务 module 中 `go get` **同一版本**;API 项目随后执行上述生成与测试步骤,手写项目只需依赖升级及业务回归。`v0.9.0-preview` 的变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。
升级时先安装目标版本的 CLI,再在业务 module 中 `go get` **同一版本**;API 项目随后执行上述生成与测试步骤,手写项目只需依赖升级及业务回归。`v0.9.1-preview` 的变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。
## 按需深入

View File

@@ -1,34 +1,31 @@
# v0.9.0-preview
# v0.9.1-preview
本版补齐已有 HTTP 项目的迁移接入能力:标准 Handler/Middleware 桥接、动态 JSON、路由级请求体限额与手写绑定、自定义标量转换。业务的响应格式、身份、资金事务和数据库实现仍由项目决定。
本版补齐已有项目接入的两个框架缺口:标准数据库 Connector 注入,以及应用级 HTTP 读写超时配置。TLS、数据库驱动日志、字符集/时区、迁移与业务规则继续由项目决定。
## 安装与升级
```sh
export GOPRIVATE=git.yuntuo.hk
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.0-preview
# 在业务 module 中:
go get git.yuntuo.hk/k2safe/go-omni@v0.9.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.1-preview
# 在各业务 module 中:
go get git.yuntuo.hk/k2safe/go-omni@v0.9.1-preview
```
将私有域名追加到已有 GOPRIVATE;凭据通过本机 Git 凭据管理器配置。CLI 与项目框架依赖一起升级。进入已生成的项目,再运行 `go-omni api generate --dry-run`、`go-omni api generate`、`go test ./...`。
私有仓库使用本机 Git 凭据管理器,并将 `git.yuntuo.hk` 追加到已有 `GOPRIVATE`。CLI 和项目依赖使用同一固定版本。v0.9.0-preview 生成代码继续兼容,本次无需为了这两个入口重新生成接口。
## 本次变化
## 接入变化
- `httpx.WrapHandler`、`WrapMiddleware`、`WrapFuncMiddleware` 将现有标准代码接入 App 中间件链。Raw 写入会记录真实状态和提交;已经输出的响应不再追加错误 JSON。下载/流式输出不整体缓存,保留底层支持的 HTTP 可选能力和取消信号。
- `.api` 的 `any` / `interface{}` 支持动态 JSON。数字保留为 json.Number,外围 DTO 严格校验不变。
- `@body` 按路由声明限额;内置 App 在全局中间件读取前限制 body。`@binding custom`、`@handlerMode manual` 生成保留的项目实现文件;上传、原文验签、下载可自行处理。`@produces` 描述手写成功输出的媒体类型。
- `scalar` 显式引用项目 Go 类型,调用标准 JSON/Text 方法;输入和输出各自生成 OpenAPI。普通基础类型不会自动放宽。
- 将项目已有 `sql.OpenDB(connector)` 的池创建替换为 `database.OpenConnector(ctx, connector, poolConfig)`;不要同时保留两个池。返回对象可用于 Health/InTx;需要标准库指针时取 `.DB`。
- 项目原有 `mysql.NewConnector`、独立 TLS、driver Logger、驱动读写时限、MySQL 版本/时区/排序规则校验继续保留,配置不进入核心框架。
- 将原来逐项修改 `http.Server` 的读写/空闲/头大小配置放入 `omni.HTTPConfig`,通过 `WithHTTPConfig` 或 `WithConfig` 传入。Go-only `ErrorLog` 支持 nethttp。
- 原有 `database.Open(ctx, Config)`、手动组装 `http.Server`、自定义 Runner 均可继续使用。零 HTTP 配置保留所选网络层已有默认行为;正值覆盖,负值报错。
详见 [迁移扩展指南](MIGRATION.md)、[完整契约](examples/contracts/compatibility.api)、[运行示例](examples/compatibility/main.go) 和 [历史更新](CHANGELOG.md)。
完整可复制示例:[数据库 Connector](database/README.md#connector-接入v091-preview)、[HTTP 配置及语义](transport/README.md#http-配置v091-preview)。其他迁移能力见 [迁移扩展指南](MIGRATION.md),从项目接入开始见 [README](README.md)。
## 兼容与边界
## 边界
原有 `.api`、公共响应配置和普通 Handler 可以继续使用。新生成代码最低运行时版本为本版。`Raw()` 现在返回跟踪包装器;直接断言某个具体 writer 类型的代码需改用标准接口/ResponseController。
独立实验性 `TransportHTTP2` 本次支持 IdleTimeout/MaxHeaderBytes 的配置透传,但不支持读头、读请求和写响应超时,设置这些正值会在监听前明确报错。native 与标准库服务器的超时含义不完全相同,详见上表;网络超时不等于 Handler 执行时间限制。`RunWith` 自定义 Runner 的配置由项目管理。
标准中间件须保留 incoming request context 并同步调用 next;第三方框架私有的参数读取器仍由项目适配。手写 Handler/绑定 hook 初始返回 501;实现和文档一致性由项目负责,未实现不能上线。切换 manual 回自动模式时须显式迁移并移除对应手写文件,生成器不会删除它。
新增能力没有引入 Gin 或 go-zero 依赖。不是完整 goctl 语法兼容层,不代表任何业务项目已完成迁移。HTTP/2 仍为实验功能,本轮没有重新做 Linux 长测、代理验收或云存储账号实网验收。
Connector 必须遵守驱动 context 协议,资源所有权按标准 `database/sql` 管理。有资源的 Connector 不跨池共享;启动 Ping 失败关闭池,成功后应用负责关闭。没有把 eTax/ePay 的数据库策略写进框架,也未修改这些业务项目或宣称它们已完成迁移。
## 验证入口
@@ -36,10 +33,13 @@ go get git.yuntuo.hk/k2safe/go-omni@v0.9.0-preview
GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=1 -count=1 -timeout=300s ./...
go vet ./...
go build ./...
python3 scripts/verifyCompatibilityRelease.py --version v0.9.0-preview \
--report performance/compatibilityRemote.json
python3 scripts/verifyConnector.py --local --report performance/connectorLocal.json
python3 scripts/verifyConnector.py --version v0.9.1-preview --report performance/connectorRemote.json
python3 scripts/verifyCompatibilityRelease.py --version v0.9.1-preview --report performance/connectorCompatibilityRemote.json
```
测试覆盖旧 Handler 404 与访问日志、提交后错误/panic、标准中间件状态/取消/提前写响应、nethttp/native 实际流式下载与请求体超限、multipart 临时文件清理、自定义标量精度/越界/nullable/集合,以及真实生成项目的验签与重生成保护。远程脚本从全新 module cache 安装 CLI,并检查生成项目的固定版本、校验和以及没有本地 replace。
Connector 脚本只使用本地独立 MySQL 8.4 容器,限制 1 CPU、768 MiB、128 PID,生成一次性 CA/证书及凭据,结束清理自己的容器/卷/临时文件。测试验证正确 TLS、错误 CA/主机名拒绝、每池时区/日志配置隔离、事务、关闭与失败回收。HTTP 测试使用回环地址验证慢头、慢 body、空闲连接、流式写背压及请求头上限。常规单测不依赖 Docker,显式脚本要求真实 TLS 测试通过,不能跳过。
本版封版验收通过:40 个包全量 race、独立 userService 回归,以及全新缓存的远程消费者验证。详见 [完整验收记录](performance/compatibilityAcceptance.md)。
历史发布及验收记录保留在 [CHANGELOG](CHANGELOG.md)、[v0.9.0 接入验收](performance/compatibilityAcceptance.md)。
全量 race(40 个测试包)、静态检查、构建及本地隔离验收已通过,详见 [验收记录](performance/connectorAcceptance.md)。

View File

@@ -156,16 +156,24 @@ func (a *App) Run(address ...string) error {
return a.RunContext(ctx, address...)
}
// NewServer creates a plain standard HTTP server around the application.
// NewServer creates a plain standard HTTP server with the configured HTTP options.
// Invalid HTTP options panic; RunContext/ServeContext return validation errors.
// Use RunContext or ServeContext to apply the configured transport and TLS.
func (a *App) NewServer(address string) *http.Server {
if err := a.config.HTTP.Validate(); err != nil {
panic(err)
}
if address == "" {
address = a.config.Address
if address == "" {
address = ":8080"
}
}
return &http.Server{Addr: address, Handler: a.router}
cfg := a.config.HTTP
return &http.Server{Addr: address, Handler: a.router,
ReadHeaderTimeout: cfg.ReadHeaderTimeout, ReadTimeout: cfg.ReadTimeout,
WriteTimeout: cfg.WriteTimeout, IdleTimeout: cfg.IdleTimeout,
MaxHeaderBytes: cfg.MaxHeaderBytes, ErrorLog: cfg.ErrorLog}
}
// RunContext serves until the context is canceled, then performs a graceful shutdown.

199
app/httpConfig_test.go Normal file
View File

@@ -0,0 +1,199 @@
package app_test
import (
"bufio"
"context"
"errors"
"io"
"log"
"net"
"net/http"
"strings"
"sync/atomic"
"testing"
"time"
omni "git.yuntuo.hk/k2safe/go-omni"
)
func TestHTTPConfiguredStandardServer(t *testing.T) {
cfg := omni.HTTPConfig{ReadHeaderTimeout: time.Second, ReadTimeout: 2 * time.Second, WriteTimeout: 3 * time.Second, IdleTimeout: 4 * time.Second, MaxHeaderBytes: 8192, ErrorLog: log.New(io.Discard, "", 0)}
a := omni.New(omni.WithHTTPConfig(cfg))
s := a.NewServer(":9000")
if s.ReadHeaderTimeout != cfg.ReadHeaderTimeout || s.ReadTimeout != cfg.ReadTimeout || s.WriteTimeout != cfg.WriteTimeout || s.IdleTimeout != cfg.IdleTimeout || s.MaxHeaderBytes != cfg.MaxHeaderBytes || s.ErrorLog != cfg.ErrorLog || s.Addr != ":9000" {
t.Fatalf("server options: %+v", s)
}
s = omni.New().NewServer("")
if s.ReadHeaderTimeout != 0 || s.ReadTimeout != 0 || s.WriteTimeout != 0 || s.IdleTimeout != 0 || s.MaxHeaderBytes != 0 || s.ErrorLog != nil {
t.Fatal("default behavior changed")
}
}
func TestHTTPInvalidStartupReleasesResources(t *testing.T) {
a := omni.New(omni.WithHTTPConfig(omni.HTTPConfig{ReadTimeout: -time.Second}))
called := 0
cleanupErr := errors.New("cleanup")
a.OnShutdown(func() error { called++; return cleanupErr })
err := a.RunContext(context.Background(), "bad address")
if err == nil || !strings.Contains(err.Error(), "http.readTimeout") || !errors.Is(err, cleanupErr) || called != 1 {
t.Fatalf("validation/cleanup: %v, %d", err, called)
}
defer func() {
if recover() == nil {
t.Error("NewServer accepted invalid config")
}
}()
a.NewServer("")
}
type httpConfigListener struct{ net.Listener }
func (l httpConfigListener) Accept() (net.Conn, error) {
c, err := l.Listener.Accept()
if err == nil {
if tcp, ok := c.(*net.TCPConn); ok {
_ = tcp.SetWriteBuffer(32 << 10)
}
}
return c, err
}
func httpConfigConnection(t *testing.T, transport string, cfg omni.HTTPConfig, handler omni.Handler) net.Conn {
t.Helper()
a := omni.New(omni.WithTransport(transport), omni.WithHTTPConfig(cfg), omni.WithShutdownTimeout(time.Second))
a.GET("/", handler)
a.POST("/", handler)
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- a.ServeContext(ctx, httpConfigListener{l}) }()
t.Cleanup(func() {
cancel()
select {
case err := <-done:
if err != nil {
t.Errorf("server shutdown: %v", err)
}
case <-time.After(3 * time.Second):
t.Error("server did not stop")
}
})
c, err := net.DialTimeout("tcp", l.Addr().String(), time.Second)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { c.Close() })
_ = c.SetDeadline(time.Now().Add(3 * time.Second))
return c
}
func TestHTTPConfiguredNetworkLimits(t *testing.T) {
for _, transport := range []string{omni.TransportNetHTTP, omni.TransportNative} {
t.Run(transport, func(t *testing.T) {
empty := func(c *omni.Context) error { w := c.Raw(); w.WriteHeader(204); return nil }
t.Run("slowHeaders", func(t *testing.T) {
c := httpConfigConnection(t, transport, omni.HTTPConfig{ReadHeaderTimeout: 60 * time.Millisecond}, empty)
if _, err := io.WriteString(c, "GET / HTTP/1.1\r\nHost: localhost\r\nX-Incomplete:"); err != nil {
t.Fatal(err)
}
_, err := io.ReadAll(c)
if e, ok := err.(net.Error); ok && e.Timeout() {
t.Fatal("server read-header timeout not applied")
}
})
t.Run("slowBody", func(t *testing.T) {
var accepted atomic.Bool
c := httpConfigConnection(t, transport, omni.HTTPConfig{ReadTimeout: 60 * time.Millisecond}, func(c *omni.Context) error {
_, err := io.ReadAll(c.Request().Body)
accepted.Store(err == nil)
return nil
})
if _, err := io.WriteString(c, "POST / HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\nContent-Length: 10\r\n\r\nx"); err != nil {
t.Fatal(err)
}
_, err := io.ReadAll(c)
if e, ok := err.(net.Error); ok && e.Timeout() {
t.Fatal("read timeout not applied")
}
if accepted.Load() {
t.Fatal("incomplete body accepted")
}
})
t.Run("idle", func(t *testing.T) {
c := httpConfigConnection(t, transport, omni.HTTPConfig{IdleTimeout: 60 * time.Millisecond}, empty)
if _, err := io.WriteString(c, "GET / HTTP/1.1\r\nHost: localhost\r\n\r\n"); err != nil {
t.Fatal(err)
}
r, err := http.ReadResponse(bufio.NewReader(c), nil)
if err != nil {
t.Fatal(err)
}
r.Body.Close()
if r.StatusCode != 204 {
t.Fatal(r.StatusCode)
}
_, err = c.Read(make([]byte, 1))
if e, ok := err.(net.Error); ok && e.Timeout() {
t.Fatal("idle timeout not applied")
}
if err == nil {
t.Fatal("idle connection not closed")
}
})
t.Run("writeBackpressure", func(t *testing.T) {
done := make(chan error, 1)
c := httpConfigConnection(t, transport, omni.HTTPConfig{WriteTimeout: 60 * time.Millisecond}, func(c *omni.Context) error {
w := c.Raw()
w.WriteHeader(200)
if err := http.NewResponseController(w).Flush(); err != nil {
done <- err
return nil
}
_, err := io.Copy(w, io.LimitReader(repeatingHTTPReader{}, 64<<20))
done <- err
return nil
})
if tcp, ok := c.(*net.TCPConn); ok {
_ = tcp.SetReadBuffer(1024)
}
if _, err := io.WriteString(c, "GET / HTTP/1.1\r\nHost: localhost\r\n\r\n"); err != nil {
t.Fatal(err)
}
select {
case err := <-done:
if err == nil {
t.Fatal("write unexpectedly completed")
}
case <-time.After(2 * time.Second):
t.Fatal("write timeout not applied")
}
})
t.Run("headerLimit", func(t *testing.T) {
c := httpConfigConnection(t, transport, omni.HTTPConfig{MaxHeaderBytes: 1024}, empty)
if _, err := io.WriteString(c, "GET / HTTP/1.1\r\nHost: localhost\r\nX-Large: "+strings.Repeat("a", 8192)+"\r\n\r\n"); err != nil {
t.Fatal(err)
}
r, err := http.ReadResponse(bufio.NewReader(c), nil)
if err != nil {
t.Fatal(err)
}
r.Body.Close()
if r.StatusCode != 431 {
t.Fatalf("oversized header status: %d", r.StatusCode)
}
})
})
}
}
type repeatingHTTPReader struct{}
func (repeatingHTTPReader) Read(b []byte) (int, error) {
for i := range b {
b[i] = 'x'
}
return len(b), nil
}

View File

@@ -57,6 +57,22 @@ func (a *App) prepareTransport() (func(context.Context, net.Listener) error, err
case config.TransportNative:
server := native.New(a)
server.ShutdownTimeout = shutdownTimeout
cfg := a.config.HTTP
if cfg.ReadHeaderTimeout > 0 {
server.ReadHeaderTimeout = cfg.ReadHeaderTimeout
}
if cfg.ReadTimeout > 0 {
server.ReadTimeout = cfg.ReadTimeout
}
if cfg.WriteTimeout > 0 {
server.WriteTimeout = cfg.WriteTimeout
}
if cfg.IdleTimeout > 0 {
server.IdleTimeout = cfg.IdleTimeout
}
if cfg.MaxHeaderBytes > 0 {
server.MaxHeaderBytes = cfg.MaxHeaderBytes
}
if tlsConfig != nil {
return func(ctx context.Context, listener net.Listener) error {
return server.ServeTLS(ctx, listener, tlsConfig)
@@ -66,6 +82,12 @@ func (a *App) prepareTransport() (func(context.Context, net.Listener) error, err
case config.TransportHTTP2:
server := h2transport.New(a)
server.ShutdownTimeout = shutdownTimeout
if a.config.HTTP.IdleTimeout > 0 {
server.IdleTimeout = a.config.HTTP.IdleTimeout
}
if a.config.HTTP.MaxHeaderBytes > 0 {
server.MaxHeaderBytes = a.config.HTTP.MaxHeaderBytes
}
return func(ctx context.Context, listener net.Listener) error {
return server.ServeTLS(ctx, listener, tlsConfig)
}, nil

View File

@@ -75,8 +75,8 @@ Handler / Service / Repository / Model 与模块注册入口。模块名为 lowe
从远程安装,生成一个带 Handler / Service / Repository / Model 分层的服务骨架:
```sh
GOPRIVATE=git.yuntuo.hk go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.0-preview
go-omni new --framework-version v0.9.0-preview --module example.com/billing ./billing
GOPRIVATE=git.yuntuo.hk go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.9.1-preview
go-omni new --framework-version v0.9.1-preview --module example.com/billing ./billing
cd billing
go mod tidy
go run ./cmd/server

View File

@@ -10,7 +10,7 @@ const frameworkModule = "git.yuntuo.hk/k2safe/go-omni"
// Source builds can carry an unpublished VCS version (including +dirty).
// Keep this fallback in sync with the next release; installed CLIs use their
// downloaded module version, which also has a module checksum.
const defaultFrameworkVersion = "v0.9.0-preview"
const defaultFrameworkVersion = "v0.9.1-preview"
// This target is not a claim that the release already exists. Source users
// select a local framework checkout until the module templates are published.

View File

@@ -28,6 +28,7 @@ type Config struct {
Address string `json:"address"`
Transport string `json:"transport"`
TLS TLSConfig `json:"tls"`
HTTP HTTPConfig `json:"http"`
RequestIDHeader string `json:"requestIdHeader"`
EnableLogger bool `json:"enableLogger"`
SlowRequestThreshold time.Duration `json:"slowRequestThreshold"`
@@ -74,11 +75,20 @@ func (c Config) Validate() error {
// ValidateTransport checks built-in transport selection without opening files
// or sockets. Empty Transport preserves the standard net/http default.
func (c Config) ValidateTransport() error {
if err := c.HTTP.Validate(); err != nil {
return err
}
switch c.Transport {
case "", TransportNetHTTP, TransportNative, TransportHTTP2:
default:
return &ConfigError{Field: "transport", Message: "must be nethttp, native or http2"}
}
if c.Transport == TransportHTTP2 && (c.HTTP.ReadHeaderTimeout > 0 || c.HTTP.ReadTimeout > 0 || c.HTTP.WriteTimeout > 0) {
return &ConfigError{Field: "http", Message: "readHeaderTimeout, readTimeout and writeTimeout are not supported by the experimental http2 transport"}
}
if c.HTTP.ErrorLog != nil && c.Transport != "" && c.Transport != TransportNetHTTP {
return &ConfigError{Field: "http.errorLog", Message: "is supported only by the nethttp transport"}
}
cert, key := strings.TrimSpace(c.TLS.CertFile), strings.TrimSpace(c.TLS.KeyFile)
if (cert == "") != (key == "") {
return &ConfigError{Field: "tls", Message: "requires both certFile and keyFile"}

View File

@@ -48,6 +48,28 @@ func applyEnv(cfg Config, prefix string) (Config, error) {
cfg.Log.Environment = envString(prefix+"ENVIRONMENT", cfg.Log.Environment)
var err error
for _, field := range []struct {
name string
value *time.Duration
}{
{"HTTP_READ_HEADER_TIMEOUT", &cfg.HTTP.ReadHeaderTimeout},
{"HTTP_READ_TIMEOUT", &cfg.HTTP.ReadTimeout},
{"HTTP_WRITE_TIMEOUT", &cfg.HTTP.WriteTimeout},
{"HTTP_IDLE_TIMEOUT", &cfg.HTTP.IdleTimeout},
} {
if value := strings.TrimSpace(os.Getenv(prefix + field.name)); value != "" {
*field.value, err = time.ParseDuration(value)
if err != nil {
return Config{}, fmt.Errorf("%s%s must be a duration: %w", prefix, field.name, err)
}
}
}
if value := strings.TrimSpace(os.Getenv(prefix + "HTTP_MAX_HEADER_BYTES")); value != "" {
cfg.HTTP.MaxHeaderBytes, err = strconv.Atoi(value)
if err != nil {
return Config{}, fmt.Errorf("%sHTTP_MAX_HEADER_BYTES must be an integer: %w", prefix, err)
}
}
if cfg.EnableLogger, err = envBool(prefix+"ENABLE_LOGGER", cfg.EnableLogger); err != nil {
return Config{}, err
}

45
config/http.go Normal file
View File

@@ -0,0 +1,45 @@
package config
import (
"log"
"time"
)
// HTTPConfig controls built-in HTTP servers. Positive values override the
// selected transport's defaults; zero keeps its existing behavior. Timeouts
// follow that transport's semantics, not a handler execution deadline.
type HTTPConfig struct {
ReadHeaderTimeout time.Duration `json:"readHeaderTimeout"`
ReadTimeout time.Duration `json:"readTimeout"`
WriteTimeout time.Duration `json:"writeTimeout"`
IdleTimeout time.Duration `json:"idleTimeout"`
MaxHeaderBytes int `json:"maxHeaderBytes"`
// ErrorLog is a Go-only hook for the nethttp server's internal diagnostics.
ErrorLog *log.Logger `json:"-"`
}
func (c HTTPConfig) Validate() error {
for _, field := range []struct {
name string
value time.Duration
}{
{"readHeaderTimeout", c.ReadHeaderTimeout},
{"readTimeout", c.ReadTimeout},
{"writeTimeout", c.WriteTimeout},
{"idleTimeout", c.IdleTimeout},
} {
if field.value < 0 {
return &ConfigError{Field: "http." + field.name, Message: "must be nonnegative"}
}
}
if c.MaxHeaderBytes < 0 {
return &ConfigError{Field: "http.maxHeaderBytes", Message: "must be nonnegative"}
}
return nil
}
// WithHTTPConfig sets the built-in server's HTTP options. RunWith uses a
// caller-owned Runner and does not apply these options to that runner.
func WithHTTPConfig(settings HTTPConfig) Option {
return func(cfg *Config) { cfg.HTTP = settings }
}

87
config/http_test.go Normal file
View File

@@ -0,0 +1,87 @@
package config_test
import (
"io"
"log"
"os"
"path/filepath"
"strings"
"testing"
"time"
"git.yuntuo.hk/k2safe/go-omni/config"
)
func TestHTTPFileAndEnvironment(t *testing.T) {
for _, file := range []struct{ name, body string }{
{"server.json", `{"http":{"readHeaderTimeout":"2s","readTimeout":"3s","writeTimeout":"4s","idleTimeout":"5s","maxHeaderBytes":8192}}`},
{"server.yaml", "http:\n readHeaderTimeout: 2s\n readTimeout: 3s\n writeTimeout: 4s\n idleTimeout: 5s\n maxHeaderBytes: 8192\n"},
} {
t.Run(file.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), file.name)
if err := os.WriteFile(path, []byte(file.body), 0600); err != nil {
t.Fatal(err)
}
cfg, err := config.LoadFile(path, "HTTP_TEST_")
if err != nil {
t.Fatal(err)
}
want := config.HTTPConfig{ReadHeaderTimeout: 2 * time.Second, ReadTimeout: 3 * time.Second, WriteTimeout: 4 * time.Second, IdleTimeout: 5 * time.Second, MaxHeaderBytes: 8192}
if cfg.HTTP != want {
t.Fatalf("HTTP: %+v", cfg.HTTP)
}
t.Setenv("HTTP_TEST_HTTP_READ_HEADER_TIMEOUT", "6s")
t.Setenv("HTTP_TEST_HTTP_READ_TIMEOUT", "7s")
t.Setenv("HTTP_TEST_HTTP_WRITE_TIMEOUT", "8s")
t.Setenv("HTTP_TEST_HTTP_IDLE_TIMEOUT", "9s")
t.Setenv("HTTP_TEST_HTTP_MAX_HEADER_BYTES", "16384")
got, err := config.ApplyEnv(cfg, "HTTP_TEST_")
if err != nil {
t.Fatal(err)
}
want = config.HTTPConfig{ReadHeaderTimeout: 6 * time.Second, ReadTimeout: 7 * time.Second, WriteTimeout: 8 * time.Second, IdleTimeout: 9 * time.Second, MaxHeaderBytes: 16384}
if got.HTTP != want {
t.Fatalf("overlay: %+v", got.HTTP)
}
})
}
}
func TestHTTPInvalidAndUnsupportedSettings(t *testing.T) {
for _, key := range []string{"READ_HEADER_TIMEOUT", "READ_TIMEOUT", "WRITE_TIMEOUT", "IDLE_TIMEOUT", "MAX_HEADER_BYTES"} {
for _, value := range []string{"invalid", "-1"} {
t.Run(key+value, func(t *testing.T) {
if value == "-1" && key != "MAX_HEADER_BYTES" {
value = "-1s"
}
t.Setenv("HTTP_INVALID_HTTP_"+key, value)
if _, err := config.LoadFromEnv("HTTP_INVALID_"); err == nil {
t.Fatal("invalid HTTP value accepted")
}
})
}
}
for _, http := range []config.HTTPConfig{{ReadHeaderTimeout: time.Second}, {ReadTimeout: time.Second}, {WriteTimeout: time.Second}} {
cfg := config.Default()
cfg.Transport = config.TransportHTTP2
cfg.TLS = config.TLSConfig{CertFile: "c", KeyFile: "k"}
cfg.HTTP = http
if err := cfg.ValidateTransport(); err == nil || !strings.Contains(err.Error(), "not supported") {
t.Fatalf("HTTP/2: %v", err)
}
}
cfg := config.Default()
cfg.Transport = config.TransportHTTP2
cfg.TLS = config.TLSConfig{CertFile: "c", KeyFile: "k"}
cfg.HTTP = config.HTTPConfig{IdleTimeout: time.Minute, MaxHeaderBytes: 8192}
if err := cfg.ValidateTransport(); err != nil {
t.Fatal(err)
}
for _, transport := range []string{config.TransportNative, config.TransportHTTP2} {
cfg.Transport = transport
cfg.HTTP = config.HTTPConfig{ErrorLog: log.New(io.Discard, "", 0)}
if err := cfg.ValidateTransport(); err == nil || !strings.Contains(err.Error(), "http.errorLog") {
t.Fatalf("ErrorLog silently ignored: %v", err)
}
}
}

View File

@@ -3,7 +3,7 @@
`database` 包提供基于标准库 `database/sql` 的连接池生命周期,不绑定 ORM 或驱动:
- `Config` / `ConfigFromEnv`:连接池参数和 `GO_OMNI_DB_*` 环境变量。
- `Open`:创建连接池、设置连接上限并在返回前执行 `PingContext`。
- `Open` / `OpenConnector`:通过 DSN 或标准 Connector 创建连接池、设置连接上限并在返回前执行 `PingContext`。
- `Health`:用于就绪检查,等待连接和 Ping 均受 `PingTimeout` 限制。
- `InTx`:统一事务提交和回滚;工作函数返回错误或 panic 时回滚。
- `Close`:由嵌入的 `*sql.DB` 提供,应用退出时显式调用。
@@ -40,7 +40,7 @@ GO_OMNI_DB_CONN_MAX_IDLE_TIME=5m
GO_OMNI_DB_PING_TIMEOUT=5s
```
`DSN` 必填,驱动默认是 `mysql`。数据库建库、迁移、表结构和具体 Repository 属于业务项目或可选模块,不会被核心包自动执行。
使用 `ConfigFromEnv` / `Open` 时 `DSN` 必填,环境变量入口的驱动默认是 `mysql`。数据库建库、迁移、表结构和具体 Repository 属于业务项目或可选模块,不会被核心包自动执行。
`MaxIdleConns=0` 会禁用空闲连接。初始化 Ping 失败会关闭已创建的连接池。
@@ -64,3 +64,61 @@ python3 scripts/verifyDatabase.py --version v0.2.0
`Config` 的 JSON/YAML 字段使用 lowerCamelCase,`DSN` 使用 `json:"-"`,连接密钥从环境变量加载。
API 脚手架 `--database mysql` 使用此流程,并要求正的 `maxOpenConns`,提供数据库 readiness 与幂等清理。
具体表结构、迁移和 Repository 留在业务项目,参考 `examples/userService`。
## Connector 接入(v0.9.1-preview)
项目已有 `database/sql/driver.Connector` 时使用 `OpenConnector`。这是标准驱动接口,核心包不增加具体驱动依赖。项目可以在同一进程创建多个使用不同证书、日志、拨号策略的连接池。
下面是 MySQL 的完整组装函数;连接地址、凭据、TLS 和驱动 Logger 都从调用方传入,不在框架中约定:
```go
package connection
import (
"context"
"crypto/tls"
"github.com/go-sql-driver/mysql"
"git.yuntuo.hk/k2safe/go-omni/database"
)
func Open(ctx context.Context, address, user, password, name string,
tlsConfig *tls.Config, logger mysql.Logger,
pool database.PoolConfig) (*database.DB, error) {
cfg := mysql.NewConfig()
cfg.Net, cfg.Addr = "tcp", address
cfg.User, cfg.Passwd, cfg.DBName = user, password, name
cfg.TLS = tlsConfig // 项目设置 RootCAs、ServerName、MinVersion 等
if logger != nil {
cfg.Logger = logger // 项目现有的 Print(...any) 日志适配器
}
connector, err := mysql.NewConnector(cfg)
if err != nil {
return nil, err
}
return database.OpenConnector(ctx, connector, pool)
}
```
调用时以 `database.DefaultPoolConfig()` 为起点,按项目需要调整。已有 `database.Config` 可调用 `.PoolConfig()` 提取池设置;`ConfigFromEnv` / `ApplyEnv` 仍是 DSN 入口,不用为 Connector 填假 DSN。`PoolConfig` 支持通过 `config.DecodeFile` 加载项目的 JSON/YAML 嵌套配置,环境覆盖由项目按自己的驱动配置组织。
| PoolConfig 字段 | 默认值 / 含义 |
| --- | --- |
| `MaxOpenConns` | `10`;`0` 表示不限制 |
| `MaxIdleConns` | `5`;`0` 禁止保留空闲连接 |
| `ConnMaxLifetime` | `30m`;`0` 不因连接年龄回收 |
| `ConnMaxIdleTime` | `5m`;`0` 不因空闲时长回收 |
| `PingTimeout` | `5s`;必须为正,约束启动 Ping 和 Health 等待 |
返回的 `*database.DB` 嵌入 `*sql.DB`,可使用 `QueryContext`、`ExecContext`、`Stats`,以及框架的 `Health`、`InTx`。需要标准数据库指针的 ORM/迁移器可取 `db.DB`;不要重复建立池。项目负责驱动支持的 context 取消与读写超时设置,框架不能强制中断不遵守 context 的自定义驱动。
**生命周期:** nil/带类型的 nil Connector、非法池参数、启动前已取消的 context 会在创建池前返回,Connector 仍归调用方。进入 `sql.OpenDB` 后由池持有;启动 Ping 失败自动关闭,成功后由调用方负责 `db.Close`,通常注册 `app.OnShutdown(db.Close)`。若 Connector 实现 `io.Closer`,遵循标准 `database/sql` 规则随池关闭;有资源的 Connector 不应跨多个池共享。资源初始化过程中、注册退出回调之前若业务校验失败,项目也须关闭池。
`db.PoolConfig()` 返回池配置快照。Connector 入口的 `db.Config()` 中 `Driver` / `DSN` 为空,不能用它重建 Connector。TLS、Logger、驱动版本检查、字符集、时区以及迁移规则保持由项目负责;不写入框架配置快照。
验收脚本使用独立本地 MySQL 8.4 容器(1 CPU / 768 MiB),自动生成测试 CA/证书与随机凭据,检查 TLS 信任/主机名、两池配置隔离、驱动日志、事务和 HTTP 超时,结束后清理本次资源:
```sh
python3 scripts/verifyConnector.py --local --report performance/connectorLocal.json
# 发布后通过全新 module cache 验证远程依赖,无本地 replace:
python3 scripts/verifyConnector.py --version v0.9.1-preview --report performance/connectorRemote.json
```

82
database/connector.go Normal file
View File

@@ -0,0 +1,82 @@
package database
import (
"context"
"database/sql"
"database/sql/driver"
"errors"
"reflect"
"time"
)
// PoolConfig contains only driver-independent settings. Connection credentials,
// TLS, logging and dial behavior belong to the application's Connector.
// Start from DefaultPoolConfig; zero PingTimeout is invalid.
type PoolConfig struct {
MaxOpenConns int `json:"maxOpenConns"`
MaxIdleConns int `json:"maxIdleConns"`
ConnMaxLifetime time.Duration `json:"connMaxLifetime"`
ConnMaxIdleTime time.Duration `json:"connMaxIdleTime"`
PingTimeout time.Duration `json:"pingTimeout"`
}
func DefaultPoolConfig() PoolConfig {
return PoolConfig{MaxOpenConns: 10, MaxIdleConns: 5, ConnMaxLifetime: 30 * time.Minute, ConnMaxIdleTime: 5 * time.Minute, PingTimeout: 5 * time.Second}
}
func (c PoolConfig) Validate() error {
if c.MaxOpenConns < 0 || c.MaxIdleConns < 0 {
return errors.New("database pool limits cannot be negative")
}
if c.MaxOpenConns > 0 && c.MaxIdleConns > c.MaxOpenConns {
return errors.New("max idle connections cannot exceed max open connections")
}
if c.ConnMaxLifetime < 0 || c.ConnMaxIdleTime < 0 || c.PingTimeout <= 0 {
return errors.New("database timeouts are invalid")
}
return nil
}
func (c PoolConfig) config() Config {
return Config{MaxOpenConns: c.MaxOpenConns, MaxIdleConns: c.MaxIdleConns, ConnMaxLifetime: c.ConnMaxLifetime, ConnMaxIdleTime: c.ConnMaxIdleTime, PingTimeout: c.PingTimeout}
}
// OpenConnector opens an independent pool using the supplied standard Connector.
// No registered driver name or DSN is required. It applies cfg and completes a
// startup Ping bounded by ctx and PingTimeout before returning.
//
// Validation failures and an already-canceled ctx leave the Connector untouched.
// Once sql.OpenDB is called, the pool owns its lifecycle: a failed startup closes
// the pool; on success the caller must Close the returned DB. As in database/sql,
// this also closes a Connector that implements io.Closer. Use a separate Connector
// for each pool if it owns resources. Nil ctx means context.Background.
func OpenConnector(ctx context.Context, connector driver.Connector, cfg PoolConfig) (*DB, error) {
if nilConnector(connector) {
return nil, errors.New("database connector is required")
}
if err := cfg.Validate(); err != nil {
return nil, err
}
if ctx == nil {
ctx = context.Background()
}
if err := ctx.Err(); err != nil {
return nil, err
}
return initializePool(ctx, sql.OpenDB(connector), cfg.config())
}
func nilConnector(connector driver.Connector) bool {
if connector == nil {
return true
}
v := reflect.ValueOf(connector)
switch v.Kind() {
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
return v.IsNil()
}
return false
}
// PoolConfig returns a copy of the settings used to create this pool.
func (db *DB) PoolConfig() PoolConfig { return db.config.PoolConfig() }

336
database/connector_test.go Normal file
View File

@@ -0,0 +1,336 @@
package database_test
import (
"context"
"database/sql"
"database/sql/driver"
"errors"
"io"
"sync"
"sync/atomic"
"testing"
"time"
"git.yuntuo.hk/k2safe/go-omni/database"
)
type fixtureConnector struct {
identity string
connects, closedConnections, closedConnector, commits, rollbacks atomic.Int32
connect func(context.Context) error
ping func(context.Context) error
query func(context.Context) error
closeErr error
}
func (c *fixtureConnector) Connect(ctx context.Context) (driver.Conn, error) {
c.connects.Add(1)
if c.connect != nil {
if err := c.connect(ctx); err != nil {
return nil, err
}
}
return &fixtureConn{owner: c}, nil
}
func (c *fixtureConnector) Driver() driver.Driver { return fixtureDriver{} }
func (c *fixtureConnector) Close() error { c.closedConnector.Add(1); return c.closeErr }
type fixtureDriver struct{}
func (fixtureDriver) Open(string) (driver.Conn, error) {
return nil, errors.New("unexpected driver name/DSN path")
}
type fixtureConn struct{ owner *fixtureConnector }
func (c *fixtureConn) Prepare(string) (driver.Stmt, error) { return nil, errors.New("not used") }
func (c *fixtureConn) Close() error { c.owner.closedConnections.Add(1); return nil }
func (c *fixtureConn) Begin() (driver.Tx, error) { return fixtureTx{c.owner}, nil }
func (c *fixtureConn) Ping(ctx context.Context) error {
if c.owner.ping != nil {
return c.owner.ping(ctx)
}
return nil
}
func (c *fixtureConn) QueryContext(ctx context.Context, _ string, _ []driver.NamedValue) (driver.Rows, error) {
if c.owner.query != nil {
if err := c.owner.query(ctx); err != nil {
return nil, err
}
}
return &fixtureRows{identity: c.owner.identity}, nil
}
type fixtureTx struct{ owner *fixtureConnector }
func (tx fixtureTx) Commit() error { tx.owner.commits.Add(1); return nil }
func (tx fixtureTx) Rollback() error { tx.owner.rollbacks.Add(1); return nil }
type fixtureRows struct {
identity string
sent bool
}
func (r *fixtureRows) Columns() []string { return []string{"identity"} }
func (r *fixtureRows) Close() error { return nil }
func (r *fixtureRows) Next(values []driver.Value) error {
if r.sent {
return io.EOF
}
r.sent = true
values[0] = r.identity
return nil
}
func TestConnectorIndependentPoolsAndContext(t *testing.T) {
type key struct{}
ctx := context.WithValue(context.Background(), key{}, "startup")
cfg := database.DefaultPoolConfig()
cfg.MaxOpenConns = 2
cfg.MaxIdleConns = 2
var owners []*fixtureConnector
var pools []*database.DB
for _, identity := range []string{"alpha", "beta"} {
owner := &fixtureConnector{identity: identity}
// Startup Connect/Ping and each query preserve their caller context.
// database/sql may also open replacement connections in the background.
owner.connect = func(ctx context.Context) error {
if owner.connects.Load() == 1 && ctx.Value(key{}) != "startup" {
return errors.New("lost request context")
}
return nil
}
owner.ping = func(ctx context.Context) error {
if ctx.Value(key{}) != "startup" {
return errors.New("lost ping context")
}
return nil
}
owner.query = func(ctx context.Context) error {
if ctx.Value(key{}) != "startup" {
return errors.New("lost query context")
}
return nil
}
pool, err := database.OpenConnector(ctx, owner, cfg)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { pool.Close() })
if pool.PoolConfig() != cfg || pool.Config().Driver != "" || pool.Config().DSN != "" || pool.Stats().MaxOpenConnections != 2 {
t.Fatal("pool settings or DSN boundary")
}
owners = append(owners, owner)
pools = append(pools, pool)
}
var wg sync.WaitGroup
for i, pool := range pools {
for n := 0; n < 16; n++ {
wg.Add(1)
go func(i int, pool *database.DB) {
defer wg.Done()
var identity string
if err := pool.QueryRowContext(ctx, "identity").Scan(&identity); err != nil {
t.Error(err)
} else if identity != owners[i].identity {
t.Error("crossed connectors")
}
}(i, pool)
}
}
wg.Wait()
for i, pool := range pools {
if pool.Stats().OpenConnections > cfg.MaxOpenConns {
t.Fatal("pool cap not applied")
}
if err := pool.Close(); err != nil {
t.Fatal(err)
}
if err := pool.Close(); err != nil {
t.Fatal(err)
}
if owners[i].closedConnector.Load() != 1 || owners[i].closedConnections.Load() != owners[i].connects.Load() {
t.Fatal("close did not release owned resources exactly once")
}
if pool.Health(ctx) == nil {
t.Fatal("closed pool is healthy")
}
}
}
func TestConnectorRejectsInvalidInputWithoutTakingOwnership(t *testing.T) {
var typedNil *fixtureConnector
for _, connector := range []driver.Connector{nil, typedNil} {
if db, err := database.OpenConnector(nil, connector, database.DefaultPoolConfig()); err == nil || db != nil {
t.Fatal("nil connector accepted")
}
}
for _, mutate := range []func(*database.PoolConfig){
func(c *database.PoolConfig) { c.MaxOpenConns = -1 }, func(c *database.PoolConfig) { c.MaxIdleConns = -1 },
func(c *database.PoolConfig) { c.MaxIdleConns = c.MaxOpenConns + 1 }, func(c *database.PoolConfig) { c.PingTimeout = 0 },
func(c *database.PoolConfig) { c.ConnMaxLifetime = -1 }, func(c *database.PoolConfig) { c.ConnMaxIdleTime = -1 },
} {
cfg := database.DefaultPoolConfig()
mutate(&cfg)
owner := &fixtureConnector{}
if db, err := database.OpenConnector(nil, owner, cfg); err == nil || db != nil {
t.Fatal("invalid config accepted")
}
if owner.connects.Load() != 0 || owner.closedConnector.Load() != 0 {
t.Fatal("validation consumed connector")
}
}
owner := &fixtureConnector{}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if db, err := database.OpenConnector(ctx, owner, database.DefaultPoolConfig()); !errors.Is(err, context.Canceled) || db != nil {
t.Fatalf("pre-canceled context: %v", err)
}
if owner.connects.Load() != 0 || owner.closedConnector.Load() != 0 {
t.Fatal("pre-cancellation consumed connector")
}
if err := database.DefaultConfig().Validate(); err == nil {
t.Fatal("legacy DSN requirements changed")
}
}
func TestConnectorStartupFailuresReleaseResources(t *testing.T) {
failure := errors.New("fixture unavailable")
closeFailure := errors.New("fixture close failed")
for _, phase := range []string{"connect", "ping"} {
t.Run(phase, func(t *testing.T) {
owner := &fixtureConnector{closeErr: closeFailure}
if phase == "connect" {
owner.connect = func(context.Context) error { return failure }
} else {
owner.ping = func(context.Context) error { return failure }
}
pool, err := database.OpenConnector(nil, owner, database.DefaultPoolConfig())
if pool != nil || !errors.Is(err, failure) || !errors.Is(err, closeFailure) {
t.Fatalf("lost failure or cleanup error: %v", err)
}
expectedConnections := int32(0)
if phase == "ping" {
expectedConnections = 1
}
if owner.closedConnector.Load() != 1 || owner.closedConnections.Load() != expectedConnections {
t.Fatal("startup failure leaked resources")
}
})
}
}
func TestConnectorTimeoutBoundsConnectAndPing(t *testing.T) {
for _, phase := range []string{"connect", "ping"} {
t.Run(phase, func(t *testing.T) {
owner := &fixtureConnector{}
cfg := database.DefaultPoolConfig()
cfg.PingTimeout = 20 * time.Millisecond
wait := func(ctx context.Context) error {
if _, ok := ctx.Deadline(); !ok {
return errors.New("missing deadline")
}
<-ctx.Done()
return ctx.Err()
}
if phase == "connect" {
owner.connect = wait
} else {
owner.ping = wait
}
pool, err := database.OpenConnector(context.Background(), owner, cfg)
if pool != nil || !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("%s timeout: %v", phase, err)
}
if owner.closedConnector.Load() != 1 {
t.Fatal("timeout leaked connector")
}
})
}
}
func TestConnectorHealthBoundsPoolWaitAndRecovers(t *testing.T) {
cfg := database.DefaultPoolConfig()
cfg.MaxOpenConns = 1
cfg.MaxIdleConns = 1
cfg.PingTimeout = 20 * time.Millisecond
owner := &fixtureConnector{}
pool, err := database.OpenConnector(nil, owner, cfg)
if err != nil {
t.Fatal(err)
}
defer pool.Close()
conn, err := pool.Conn(context.Background())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
if err = pool.Health(nil); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("waiting for busy pool was not bounded: %v", err)
}
if err = conn.Close(); err != nil {
t.Fatal(err)
}
if err = pool.Health(nil); err != nil {
t.Fatal(err)
}
}
func TestConnectorIdleZeroAndLifetime(t *testing.T) {
for _, mode := range []string{"noIdle", "expired"} {
t.Run(mode, func(t *testing.T) {
cfg := database.DefaultPoolConfig()
if mode == "noIdle" {
cfg.MaxIdleConns = 0
} else {
cfg.ConnMaxLifetime = time.Nanosecond
}
owner := &fixtureConnector{}
pool, err := database.OpenConnector(nil, owner, cfg)
if err != nil {
t.Fatal(err)
}
defer pool.Close()
if err = pool.Health(nil); err != nil {
t.Fatal(err)
}
if owner.connects.Load() < 2 || owner.closedConnections.Load() == 0 {
t.Fatal("pool did not retire connections")
}
if mode == "noIdle" && pool.Stats().Idle != 0 {
t.Fatal("zero idle not honored")
}
if mode == "expired" && pool.Stats().MaxLifetimeClosed == 0 {
t.Fatal("lifetime not honored")
}
})
}
}
func TestConnectorTransactionsKeepCommitAndRollbackSemantics(t *testing.T) {
owner := &fixtureConnector{}
pool, err := database.OpenConnector(nil, owner, database.DefaultPoolConfig())
if err != nil {
t.Fatal(err)
}
defer pool.Close()
ctx := context.Background()
if err = pool.InTx(ctx, nil, func(*sql.Tx) error { return nil }); err != nil {
t.Fatal(err)
}
failed := errors.New("business failed")
if err = pool.InTx(ctx, nil, func(*sql.Tx) error { return failed }); !errors.Is(err, failed) {
t.Fatal(err)
}
func() {
defer func() {
if recover() != failed {
t.Error("panic not preserved")
}
}()
_ = pool.InTx(ctx, nil, func(*sql.Tx) error { panic(failed) })
}()
if owner.commits.Load() != 1 || owner.rollbacks.Load() != 2 {
t.Fatal("transaction semantics changed")
}
}

View File

@@ -23,13 +23,14 @@ type Config struct {
PingTimeout time.Duration `json:"pingTimeout"`
}
func DefaultConfig() Config {
return Config{
MaxOpenConns: 10,
MaxIdleConns: 5,
ConnMaxLifetime: 30 * time.Minute,
ConnMaxIdleTime: 5 * time.Minute,
PingTimeout: 5 * time.Second,
func DefaultConfig() Config { return DefaultPoolConfig().config() }
// PoolConfig returns driver-independent connection pool settings.
func (c Config) PoolConfig() PoolConfig {
return PoolConfig{
MaxOpenConns: c.MaxOpenConns, MaxIdleConns: c.MaxIdleConns,
ConnMaxLifetime: c.ConnMaxLifetime, ConnMaxIdleTime: c.ConnMaxIdleTime,
PingTimeout: c.PingTimeout,
}
}
@@ -83,16 +84,7 @@ func (c Config) Validate() error {
if strings.TrimSpace(c.DSN) == "" {
return errors.New("database DSN is required")
}
if c.MaxOpenConns < 0 || c.MaxIdleConns < 0 {
return errors.New("database pool limits cannot be negative")
}
if c.MaxIdleConns > c.MaxOpenConns && c.MaxOpenConns > 0 {
return errors.New("max idle connections cannot exceed max open connections")
}
if c.ConnMaxLifetime < 0 || c.ConnMaxIdleTime < 0 || c.PingTimeout <= 0 {
return errors.New("database timeouts are invalid")
}
return nil
return c.PoolConfig().Validate()
}
// DB owns a connection pool and exposes the embedded database/sql methods.
@@ -112,23 +104,30 @@ func Open(ctx context.Context, cfg Config) (*DB, error) {
if err != nil {
return nil, fmt.Errorf("open database: %w", err)
}
if cfg.MaxOpenConns > 0 {
return initializePool(ctx, db, cfg)
}
// initializePool is the shared ownership boundary for both creation paths.
// All configuration is validated before creating the sql.DB.
func initializePool(ctx context.Context, db *sql.DB, cfg Config) (*DB, error) {
db.SetMaxOpenConns(cfg.MaxOpenConns)
}
// database/sql uses zero to disable idle connections; do not leave its
// default pool size in place when the caller explicitly chooses zero.
db.SetMaxIdleConns(cfg.MaxIdleConns)
db.SetMaxIdleConns(cfg.MaxIdleConns) // Zero explicitly disables idle connections.
db.SetConnMaxLifetime(cfg.ConnMaxLifetime)
db.SetConnMaxIdleTime(cfg.ConnMaxIdleTime)
checkCtx, cancel := context.WithTimeout(ctx, cfg.PingTimeout)
defer cancel()
if err := db.PingContext(checkCtx); err != nil {
_ = db.Close()
return nil, fmt.Errorf("ping database: %w", err)
pingErr := fmt.Errorf("ping database: %w", err)
if closeErr := db.Close(); closeErr != nil {
return nil, errors.Join(pingErr, fmt.Errorf("close database after failed ping: %w", closeErr))
}
return nil, pingErr
}
return &DB{DB: db, config: cfg}, nil
}
// Config returns the original DSN configuration. Pools opened with a Connector
// have empty Driver/DSN fields; their connector cannot be reconstructed from Config.
func (db *DB) Config() Config { return db.config }
func (db *DB) Health(ctx context.Context) error {

View File

@@ -0,0 +1,180 @@
package database_test
import (
"context"
"crypto/tls"
"crypto/x509"
"database/sql"
"errors"
"net"
"os"
"sync/atomic"
"testing"
"time"
"git.yuntuo.hk/k2safe/go-omni/database"
"github.com/go-sql-driver/mysql"
)
type connectorDriverLog struct{ count atomic.Int32 }
func (l *connectorDriverLog) Print(...any) { l.count.Add(1) }
func TestMySQLConnectorDriverLoggerIsolation(t *testing.T) {
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
stopped := make(chan struct{})
go func() {
defer close(stopped)
for {
c, err := listener.Accept()
if err != nil {
return
}
c.Close()
}
}()
defer func() { listener.Close(); <-stopped }()
logs := [2]connectorDriverLog{}
for i := range logs {
cfg := mysql.NewConfig()
cfg.Net = "tcp"
cfg.Addr = listener.Addr().String()
cfg.Timeout = time.Second
cfg.ReadTimeout = time.Second
cfg.Logger = &logs[i]
connector, err := mysql.NewConnector(cfg)
if err != nil {
t.Fatal(err)
}
before := [2]int32{logs[0].count.Load(), logs[1].count.Load()}
db, err := database.OpenConnector(context.Background(), connector, database.DefaultPoolConfig())
if err == nil {
db.Close()
t.Fatal("handshake to closed socket succeeded")
}
if logs[i].count.Load() <= before[i] || logs[1-i].count.Load() != before[1-i] {
t.Fatal("driver logger did not stay on its connector")
}
}
}
func TestMySQLConnectorTLSAndPoolIsolation(t *testing.T) {
if os.Getenv("GO_OMNI_CONNECTOR_MYSQL_TEST") != "1" {
t.Skip("use scripts/verifyConnector.py for isolated MySQL TLS verification")
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
ca, err := os.ReadFile(os.Getenv("GO_OMNI_CONNECTOR_CA_FILE"))
if err != nil {
t.Fatal(err)
}
roots := x509.NewCertPool()
if !roots.AppendCertsFromPEM(ca) {
t.Fatal("invalid test CA")
}
cfg := mysql.NewConfig()
cfg.Net = "tcp"
cfg.Addr = os.Getenv("GO_OMNI_CONNECTOR_ADDR")
cfg.User = os.Getenv("GO_OMNI_CONNECTOR_USER")
cfg.Passwd = os.Getenv("GO_OMNI_CONNECTOR_PASSWORD")
cfg.DBName = os.Getenv("GO_OMNI_CONNECTOR_DATABASE")
cfg.Timeout = time.Second
cfg.ReadTimeout = 2 * time.Second
cfg.WriteTimeout = 2 * time.Second
cfg.TLS = &tls.Config{MinVersion: tls.VersionTLS12, ServerName: "localhost", RootCAs: roots}
pool := database.DefaultPoolConfig()
pool.MaxOpenConns = 2
pool.MaxIdleConns = 2
pool.PingTimeout = 2 * time.Second
open := func(cfg *mysql.Config) (*database.DB, error) {
connector, err := mysql.NewConnector(cfg)
if err != nil {
return nil, err
}
return database.OpenConnector(ctx, connector, pool)
}
cfg.Params = map[string]string{"time_zone": "'+00:00'"}
first, err := open(cfg)
if err != nil {
t.Fatal(err)
}
defer first.Close()
other := cfg.Clone()
other.Params["time_zone"] = "'+08:00'"
second, err := open(other)
if err != nil {
t.Fatal(err)
}
defer second.Close()
for i, db := range []*database.DB{first, second} {
var name, cipher, zone string
if err := db.QueryRowContext(ctx, "SHOW SESSION STATUS LIKE 'Ssl_cipher'").Scan(&name, &cipher); err != nil {
t.Fatal(err)
}
if cipher == "" {
t.Fatal("connection is not encrypted")
}
if err := db.QueryRowContext(ctx, "SELECT @@session.time_zone").Scan(&zone); err != nil {
t.Fatal(err)
}
if zone != []string{"+00:00", "+08:00"}[i] {
t.Fatalf("session config leaked: %s", zone)
}
if db.Stats().MaxOpenConnections != 2 {
t.Fatal("pool configuration ignored")
}
}
for _, bad := range []string{"untrustedCA", "wrongHostname"} {
t.Run(bad, func(t *testing.T) {
invalid := cfg.Clone()
if bad == "untrustedCA" {
invalid.TLS.RootCAs = x509.NewCertPool()
} else {
invalid.TLS.ServerName = "wrong.invalid"
}
db, err := open(invalid)
if err == nil {
db.Close()
t.Fatal("invalid server certificate accepted")
}
var verification *tls.CertificateVerificationError
if !errors.As(err, &verification) {
t.Fatalf("not a TLS verification failure: %v", err)
}
if err := first.Health(ctx); err != nil {
t.Fatalf("failed connector affected existing pool: %v", err)
}
})
}
if _, err := first.ExecContext(ctx, "CREATE TABLE connector_check (id INT PRIMARY KEY)"); err != nil {
t.Fatal(err)
}
if err := first.InTx(ctx, nil, func(tx *sql.Tx) error {
_, err := tx.ExecContext(ctx, "INSERT INTO connector_check VALUES (1)")
return err
}); err != nil {
t.Fatal(err)
}
rollbackErr := errors.New("rollback requested")
if err := first.InTx(ctx, nil, func(tx *sql.Tx) error {
if _, err := tx.ExecContext(ctx, "INSERT INTO connector_check VALUES (2)"); err != nil {
return err
}
return rollbackErr
}); !errors.Is(err, rollbackErr) {
t.Fatal(err)
}
var count int
if err := second.QueryRowContext(ctx, "SELECT COUNT(*) FROM connector_check").Scan(&count); err != nil || count != 1 {
t.Fatalf("transaction count=%d err=%v", count, err)
}
if err := first.Close(); err != nil {
t.Fatal(err)
}
if err := second.Health(ctx); err != nil {
t.Fatal("closing first pool affected second:", err)
}
}

View File

@@ -26,6 +26,7 @@ type HTTPRouterGroup = httpx.RouterGroup
type APIError = apierrors.APIError
type Config = config.Config
type TLSConfig = config.TLSConfig
type HTTPConfig = config.HTTPConfig
type Option = config.Option
type Response = response.Response
type LogConfig = logx.Config
@@ -62,6 +63,8 @@ func WithConfig(settings Config) Option { return config.WithConfig(settings) }
func WithTransport(name string) Option { return config.WithTransport(name) }
func WithHTTPConfig(settings HTTPConfig) Option { return config.WithHTTPConfig(settings) }
func WithTLS(certFile, keyFile string) Option { return config.WithTLS(certFile, keyFile) }
func WithLogConfig(settings LogConfig) Option { return config.WithLogConfig(settings) }

View File

@@ -0,0 +1,24 @@
# v0.9.1-preview Connector 与 HTTP 配置验收
## 范围
新增标准 `driver.Connector` 注入和 HTTP 服务器读写配置;没有修改 eTax/ePay 业务代码。数据库驱动/TLS/Logger/时区/版本校验仍归项目。网络测试只监听本地回环地址,MySQL 使用独立本地容器(1 CPU / 768 MiB / 128 PID),没有使用业务服务器压测。
## 全量回归
- `GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=1 -count=1 -timeout=300s ./...`:40 个有测试的包全部通过,包含真实脚手架/契约生成的集成消费者。
- `go vet ./...`、`go build ./...`、gofmt、`git diff --check` 通过。
## 专项验收
- `go test -race -count=3 ./database ./app ./config` 通过。
- 两个 Connector 并发读写保持独立;启动 Connect/Ping 与查询 context 保留,后台补连接遵循 database/sql 行为;最大连接数、禁用 idle、连接生命周期、健康检查等待上限及恢复验证通过。
- nil/带类型的 nil、无效参数、启动前取消不取得 Connector 所有权;Connect/Ping 失败与超时关闭池,清理错误和原错误可同时检查;成功关闭幂等,事务提交/错误/panic 回滚通过。
- 标准库服务器字段、ErrorLog 透传、无效配置启动前返回并清理资源、JSON/YAML/环境变量覆盖通过;不支持的独立 http2 选项明确拒绝。
- nethttp/native 的实际 TCP 慢请求头、慢 body、Keep-Alive 空闲关闭、Flush 后客户端停止读取触发写超时、超限头 431 通过。测试不假定两种网络层有完全相同的超时响应格式。
- `scripts/verifyConnector.py --local` 的独立消费者通过:真实 MySQL 8.4 TLS、错误 CA/主机名拒绝、不同 session time_zone、每个 Connector 的驱动 Logger、两池独立关闭、事务提交/回滚。所有专项测试无跳过,详情见 [本地记录](connectorLocal.json)。
- MySQL 镜像固定记录为 `sha256:1afd25fe334a1be64a3866495dd43a6bdba3c55dd1d6190ea12933163fe2843c`。临时容器、卷、证书、凭据及消费者目录已清理。
环境:`go version go1.27.1 darwin/arm64`。记录时间:2026-09-23T18:17:16.897323+00:00。
独立 `TransportHTTP2` 的逐请求读写时限未实现,本次没有进行 Linux soak、代理长测或完整业务迁移验收。

View File

@@ -0,0 +1,70 @@
{
"passed": true,
"frameworkVersion": "v0.9.1-preview",
"localSource": true,
"frameworkSum": null,
"freshModuleCache": false,
"noLocalReplace": false,
"mysqlImage": "sha256:1afd25fe334a1be64a3866495dd43a6bdba3c55dd1d6190ea12933163fe2843c",
"mysqlLimits": {
"cpus": 1,
"memoryMiB": 768,
"pids": 128
},
"durationSeconds": 9.61,
"tests": [
"TestHTTPFileAndEnvironment/server.json",
"TestHTTPFileAndEnvironment/server.yaml",
"TestHTTPFileAndEnvironment",
"TestHTTPInvalidAndUnsupportedSettings/READ_HEADER_TIMEOUTinvalid",
"TestHTTPInvalidAndUnsupportedSettings/READ_HEADER_TIMEOUT-1",
"TestHTTPInvalidAndUnsupportedSettings/READ_TIMEOUTinvalid",
"TestHTTPInvalidAndUnsupportedSettings/READ_TIMEOUT-1",
"TestHTTPInvalidAndUnsupportedSettings/WRITE_TIMEOUTinvalid",
"TestHTTPInvalidAndUnsupportedSettings/WRITE_TIMEOUT-1",
"TestHTTPInvalidAndUnsupportedSettings/IDLE_TIMEOUTinvalid",
"TestHTTPInvalidAndUnsupportedSettings/IDLE_TIMEOUT-1",
"TestHTTPInvalidAndUnsupportedSettings/MAX_HEADER_BYTESinvalid",
"TestHTTPInvalidAndUnsupportedSettings/MAX_HEADER_BYTES-1",
"TestHTTPInvalidAndUnsupportedSettings",
"TestHTTPConfiguredStandardServer",
"TestHTTPInvalidStartupReleasesResources",
"TestHTTPConfiguredNetworkLimits/nethttp/slowHeaders",
"TestHTTPConfiguredNetworkLimits/nethttp/slowBody",
"TestHTTPConfiguredNetworkLimits/nethttp/idle",
"TestHTTPConfiguredNetworkLimits/nethttp/writeBackpressure",
"TestConnectorIndependentPoolsAndContext",
"TestConnectorRejectsInvalidInputWithoutTakingOwnership",
"TestConnectorStartupFailuresReleaseResources/connect",
"TestConnectorStartupFailuresReleaseResources/ping",
"TestConnectorStartupFailuresReleaseResources",
"TestConnectorTimeoutBoundsConnectAndPing/connect",
"TestConnectorTimeoutBoundsConnectAndPing/ping",
"TestConnectorTimeoutBoundsConnectAndPing",
"TestConnectorHealthBoundsPoolWaitAndRecovers",
"TestConnectorIdleZeroAndLifetime/noIdle",
"TestConnectorIdleZeroAndLifetime/expired",
"TestConnectorIdleZeroAndLifetime",
"TestConnectorTransactionsKeepCommitAndRollbackSemantics",
"TestMySQLConnectorDriverLoggerIsolation",
"TestMySQLConnectorTLSAndPoolIsolation/untrustedCA",
"TestMySQLConnectorTLSAndPoolIsolation/wrongHostname",
"TestMySQLConnectorTLSAndPoolIsolation",
"TestHTTPConfiguredNetworkLimits/nethttp/headerLimit",
"TestHTTPConfiguredNetworkLimits/nethttp",
"TestHTTPConfiguredNetworkLimits/native/slowHeaders",
"TestHTTPConfiguredNetworkLimits/native/slowBody",
"TestHTTPConfiguredNetworkLimits/native/idle",
"TestHTTPConfiguredNetworkLimits/native/writeBackpressure",
"TestHTTPConfiguredNetworkLimits/native/headerLimit",
"TestHTTPConfiguredNetworkLimits/native",
"TestHTTPConfiguredNetworkLimits"
],
"sourceHashes": {
"database/connector_test.go": "0663dfc9794ba24130225fa3b6418a6d5f75d57d0f01686151f70c7ea3f592c0",
"database/mysqlConnector_test.go": "e1cdd11ef76b09f1332ba92bfdde5b1917d84c85cc42fac48f65a8b23f5e9720",
"app/httpConfig_test.go": "b060dc0b76fb8e5feebae7e3b12cb7bca845693fcc097e2a0496a5ba937c837a",
"config/http_test.go": "0cae3dcf213369cfaff5487bf630c2bc3ec097739ca1d40794124e69c3880db8",
"scripts/verifyConnector.py": "a0ba3fa29a4d1efcbed5dad82c09afb97d8ddecd904f20fee1e3aa74ec0f9e06"
}
}

133
scripts/verifyConnector.py Normal file
View File

@@ -0,0 +1,133 @@
#!/usr/bin/env python3
"""Verify Connector TLS/isolation and HTTP limits from a local or remote module.
Uses a disposable local MySQL container (1 CPU, 768 MiB), private test credentials,
and generated test certificates. Never connects to a project database.
"""
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import secrets
import shutil
import subprocess
import tempfile
import time
import uuid
ROOT = Path(__file__).resolve().parents[1]
MODULE = "git.yuntuo.hk/k2safe/go-omni"
SOURCES = ("database/connector_test.go", "database/mysqlConnector_test.go",
"app/httpConfig_test.go", "config/http_test.go")
def run(args, cwd, env, timeout=300):
result = subprocess.run([str(a) for a in args], cwd=cwd, env=env,
text=True, capture_output=True, timeout=timeout)
output = re.sub(r"https://[^/@\s]+:[^/@\s]+@", "https://", result.stdout + result.stderr)
# Never print generated database credentials, including from a failed command.
for key in ("MYSQL_ROOT_PASSWORD", "MYSQL_PASSWORD", "GO_OMNI_CONNECTOR_PASSWORD"):
if env.get(key):
output = output.replace(env[key], "[redacted]")
if result.returncode:
raise RuntimeError(f"{args[0]} failed ({result.returncode}): {output}")
return output
def main():
parser = argparse.ArgumentParser(description=__doc__)
source = parser.add_mutually_exclusive_group(required=True)
source.add_argument("--local", action="store_true")
source.add_argument("--version")
parser.add_argument("--report", type=Path, required=True)
parser.add_argument("--image", default="mysql:8.4")
args = parser.parse_args()
if args.version and not re.fullmatch(r"v\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?", args.version):
parser.error("a fixed release version is required")
env = {k: v for k, v in os.environ.items() if not k.startswith("GO_OMNI_")}
env.update(GOWORK="off", GOFLAGS="")
for key in ("GOPRIVATE", "GONOSUMDB"):
env[key] = ",".join(filter(None, [env.get(key), "git.yuntuo.hk"]))
began = time.monotonic()
with tempfile.TemporaryDirectory(prefix="goOmniConnector-") as directory:
temp = Path(directory)
endpoint = env.get("DOCKER_HOST") or run(["docker", "context", "inspect", "--format", "{{.Endpoints.docker.Host}}"], temp, env).strip()
if not endpoint.startswith(("unix://", "npipe://")):
raise RuntimeError("requires local Docker; remote servers are not used")
certs = temp / "certs"
certs.mkdir(mode=0o755)
run(["openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", certs / "ca.key", "-out", certs / "ca.pem", "-days", "2", "-subj", "/CN=GoOmni Test CA"], temp, env)
run(["openssl", "req", "-newkey", "rsa:2048", "-nodes", "-keyout", certs / "server.key", "-out", temp / "server.csr", "-subj", "/CN=localhost"], temp, env)
(temp / "extensions.cnf").write_text("subjectAltName=DNS:localhost,IP:127.0.0.1\nbasicConstraints=CA:FALSE\nextendedKeyUsage=serverAuth\n")
run(["openssl", "x509", "-req", "-in", temp / "server.csr", "-CA", certs / "ca.pem", "-CAkey", certs / "ca.key", "-CAcreateserial", "-out", certs / "server.pem", "-days", "2", "-extfile", temp / "extensions.cnf"], temp, env)
(certs / "ca.key").unlink()
for path in certs.iterdir():
path.chmod(0o644) # Disposable fixtures, readable by the container's mysql uid.
project = temp / "consumer"
project.mkdir()
version = args.version or "v0.9.1-preview"
(project / "go.mod").write_text(f"module example.com/connectorAcceptance\n\ngo 1.22.0\n\nrequire {MODULE} {version}\n")
if args.local:
run(["go", "mod", "edit", f"-replace={MODULE}={ROOT}"], project, env)
else:
env["GOMODCACHE"] = str(temp / "modcache")
for name in SOURCES:
target = project / name
target.parent.mkdir(exist_ok=True)
shutil.copyfile(ROOT / name, target)
print("Resolving consumer dependency", flush=True)
run(["go", "mod", "tidy"], project, env)
module = json.loads(run(["go", "list", "-m", "-json", MODULE], project, env))
if not args.local:
assert module["Version"] == args.version and module.get("Sum") and not module.get("Replace"), module
env.update(MYSQL_ROOT_PASSWORD=secrets.token_hex(24), MYSQL_PASSWORD=secrets.token_hex(24))
name = "go-omni-connector-" + uuid.uuid4().hex[:12]
created = False
try:
print("Starting isolated MySQL TLS fixture (1 CPU, 768 MiB)", flush=True)
run(["docker", "create", "--name", name, "--label", "go-omni.test=connector", "--cpus", "1", "--memory", "768m", "--pids-limit", "128", "--publish", "127.0.0.1::3306", "--mount", f"type=bind,src={certs},dst=/certs,readonly", "--env", "MYSQL_ROOT_PASSWORD", "--env", "MYSQL_PASSWORD", "--env", "MYSQL_DATABASE=omni_test", "--env", "MYSQL_USER=omni_test", args.image, "--ssl-ca=/certs/ca.pem", "--ssl-cert=/certs/server.pem", "--ssl-key=/certs/server.key", "--require-secure-transport=ON", "--innodb-buffer-pool-size=64M", "--max-connections=20"], temp, env)
created = True
run(["docker", "start", name], temp, env)
ready = False
for _ in range(60):
try:
check_env = dict(env, MYSQL_PWD=env["MYSQL_PASSWORD"])
result = run(["docker", "exec", "--env", "MYSQL_PWD", name, "mysql", "--protocol=tcp", "-h127.0.0.1", "-uomni_test", "--batch", "--skip-column-names", "-e", "SELECT 1"], temp, check_env, timeout=5)
if result.strip() == "1":
ready = True
break
except (RuntimeError, subprocess.TimeoutExpired):
pass
time.sleep(1)
if not ready:
raise RuntimeError("isolated MySQL fixture did not become ready")
addr = run(["docker", "port", name, "3306/tcp"], temp, env).strip()
assert addr.startswith("127.0.0.1:"), addr
env.update(GO_OMNI_CONNECTOR_MYSQL_TEST="1", GO_OMNI_CONNECTOR_ADDR=addr,
GO_OMNI_CONNECTOR_USER="omni_test", GO_OMNI_CONNECTOR_PASSWORD=env["MYSQL_PASSWORD"],
GO_OMNI_CONNECTOR_DATABASE="omni_test", GO_OMNI_CONNECTOR_CA_FILE=str(certs / "ca.pem"))
print("Testing TLS trust, per-connector options, driver logging, lifecycle and HTTP limits", flush=True)
output = run(["go", "test", "-race", "-count=1", "-json", "-timeout=60s", "./..."], project, env)
events = [json.loads(line) for line in output.splitlines() if line.startswith("{")]
passed = [e["Test"] for e in events if e.get("Action") == "pass" and "Test" in e]
assert "TestMySQLConnectorTLSAndPoolIsolation" in passed
assert not any(e.get("Action") in ("fail", "skip") for e in events), output
image = run(["docker", "inspect", "--format", "{{.Image}}", name], temp, env).strip()
finally:
if created:
run(["docker", "rm", "--force", "--volumes", name], temp, env)
report = {"passed": True, "frameworkVersion": version, "localSource": args.local,
"frameworkSum": module.get("Sum"), "freshModuleCache": not args.local,
"noLocalReplace": not bool(module.get("Replace")), "mysqlImage": image,
"mysqlLimits": {"cpus": 1, "memoryMiB": 768, "pids": 128},
"durationSeconds": round(time.monotonic() - began, 2), "tests": passed,
"sourceHashes": {name: hashlib.sha256((ROOT / name).read_bytes()).hexdigest() for name in (*SOURCES, "scripts/verifyConnector.py")}}
args.report.parent.mkdir(parents=True, exist_ok=True)
args.report.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n")
print("PASS; isolated container, credentials, certificates and consumer removed", flush=True)
if __name__ == "__main__":
main()

View File

@@ -14,7 +14,7 @@ registerRoutes(application) // 与 omni.New() 复用同一份业务注册代码
`transport/nethttp` 是当前内置的标准库路由实现。它提供一套可移植的路径、方法和错误回退规则;自研网络层可以直接实现同一个 `httpx.Router` 契约。
`transport/http2` 是独立的 opt-in 实验 Runner,负责连接接入、TLS/ALPN、连接上限和全局请求背压,底层帧解析与 stream 流控暂由 `golang.org/x/net/http2` 提供。它不替换 `transport/native`,也不改变默认启动器;只有通过 `App.RunWith(ctx, http2.New(nil), address)` 显式启用才会运行。
`transport/http2` 是独立的 opt-in 实验 Runner,负责连接接入、TLS/ALPN、连接上限和全局请求背压,底层帧解析与 stream 流控暂由 `golang.org/x/net/http2` 提供。它不替换 `transport/native`,也不改变默认启动器;通过 `WithTransport(TransportHTTP2)` 或 `App.RunWith(ctx, http2.New(nil), address)` 显式启用后运行。
`transport/native` 是第一版自研 HTTP/1.1 服务端:它自己管理 TCP listener、连接、Keep-Alive、请求解析、响应写回和优雅关闭,再把请求交给现有 `http.Handler`。请求体支持 `Content-Length` 和 chunked,并统一受 `MaxBodyBytes` 限制;其他传输编码会被拒绝。它仍使用标准 `http.Request` / `http.ResponseWriter` 作为业务桥接,因此不会影响现有 Handler / Service。
@@ -119,3 +119,37 @@ python3 scripts/verifyTransports.py --version v0.6.0-preview
第一条验证标准库适配器复用完整 Handler / Service / Repository。第二条在源码目录外创建无 replace 的消费模块,从远程版本导入同一套验收包;自研适配器也应使用同一个公共测试包。
当前替换边界分成两层:`transport/nethttp` 提供标准库路由,`transport/native` 进一步接管 TCP 监听、连接管理和 HTTP/1.1 协议 I/O。`App.Run` / `App.RunContext` 仍使用标准库 `net/http.Server`,需要显式调用 `App.RunWith(ctx, native.New(nil), address)` 才会启用 native 预览实现。native 当前通过标准 `http.Request` / `http.ResponseWriter` 桥接到业务层,后续可在不改 Handler / Service 的前提下逐步替换解析器和响应写回。
## HTTP 配置(v0.9.1-preview)
`omni.WithHTTPConfig` / `config.Config.HTTP` 提供服务端配置入口,作用于 `Run`、`RunContext`、`ServeContext`;也作用于 `NewServer` 创建的标准库服务器。与客户端 `httpc` 的出站请求超时无关。
```go
application := omni.New(omni.WithHTTPConfig(omni.HTTPConfig{
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 30 * time.Second,
WriteTimeout: 30 * time.Second,
IdleTimeout: 60 * time.Second,
MaxHeaderBytes: 1 << 20,
// ErrorLog: existingLogger, // *log.Logger,仅 nethttp;不通过 YAML/JSON 序列化
}))
// 注册业务路由后:
return application.RunContext(ctx, ":8080")
```
| 字段 | nethttp | native | 独立实验 http2 |
| --- | --- | --- | --- |
| `ReadHeaderTimeout` | 标准 http.Server 语义 | 请求头读取时限 | 正值拒绝启动 |
| `ReadTimeout` | 整个请求读取时限,包括头与 body | 头解析完成后读取 body 的时限 | 正值拒绝启动 |
| `WriteTimeout` | 标准 http.Server 写截止时间 | 每次底层响应写入/Flush 的截止时间 | 正值拒绝启动 |
| `IdleTimeout` | Keep-Alive 等待下一请求 | 复用连接等待下一请求(含头读取);首个请求读头也取它与读头时限的较小值 | HTTP/2 空闲连接超时 |
| `MaxHeaderBytes` | 标准 http.Server 头限制语义 | native 请求头字节限制 | 解压后请求头限制 |
| `ErrorLog` | 标准库内部错误日志,不是访问日志 | 非 nil 拒绝启动 | 非 nil 拒绝启动 |
参数为正时覆盖适配器默认值;**零值保留默认行为,不统一表示禁用**。nethttp 未配置时遵循 Go 标准服务器零值,包括读头/空闲时限回退到 ReadTimeout 的规则;native 仍默认读头 5s、读 body 30s、写 30s、空闲 60s、头 1 MiB。负数在启动前报错;HTTP/2 不支持的字段也会明确报错,不会悄悄忽略。默认 nethttp 的 TLS/HTTP2 行为仍由标准库管理,表格最后一列专指 `TransportHTTP2`。
I/O 超时不会强制终止 CPU 计算或数据库调用;业务和下游调用继续通过 `context` 控制时限。native 需要流式输出时显式 `Flush()`,否则保持原有缓冲响应方式。不要依赖不同网络层在慢请求时返回完全相同的 HTTP 错误体;有些超时会直接断开连接。
YAML/JSON 字段与上述字段对应的 lowerCamelCase 一致,位于 `config.Config` 的 `http` 对象。脚手架将框架配置包在 `server` 下,因此生成项目填写 `server.http`;直接用 `config.LoadFile` 加载框架配置时填写根级 `http`。环境变量为 `GO_OMNI_HTTP_READ_HEADER_TIMEOUT`、`GO_OMNI_HTTP_READ_TIMEOUT`、`GO_OMNI_HTTP_WRITE_TIMEOUT`、`GO_OMNI_HTTP_IDLE_TIMEOUT`(如 `30s`),以及 `GO_OMNI_HTTP_MAX_HEADER_BYTES`(整数)。`omni.New` 不自动加载环境;先 `config.LoadFile` / `config.ApplyEnv`,再 `WithConfig`。Go-only 的 `ErrorLog` 在文件加载之后设置。
`NewServer` 始终返回普通 `*http.Server`,应用 HTTP 字段但不负责加载 TLS、监听、退出或资源清理;非法 HTTP 值会 panic。`RunContext` / `ServeContext` 的非法配置则作为 error 返回并关闭 App 资源。手动修改返回服务器的字段仍然有效。项目自己的 Runner 通过 `RunWith` 接入时,配置由该 Runner 持有,这组 HTTP 配置不会覆盖它。