Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

reduce CGO calls when scanning rows #1291

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

reduce CGO calls when scanning rows #1291

wants to merge 2 commits into from

Commits on Nov 5, 2024

  1. ci: fix icu4c on macos and CIFuzz test

    @charlievieth
    charlievieth committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    821c883 View commit details
    Browse the repository at this point in the history

  2. reduce CGO calls when scanning rows 

    This commit improves the performance of queries by at least 20% by only
collecting the information needed to convert sqlite3 values to Go values
and by batching CGO calls when scanning rows (the performance
improvement scales with the number of columns being scanned).

This commit adds a new coltype field to the SQLiteRows struct which
stores the declared column type (either data/time or boolean) and the
sqlite3 datatype. Previously, this library would fetch the string
representation of each column, which is inefficient and rarely needed
since the non-standard SQLiteRows.DeclTypes method is rarely called.

It also changes the benchmark suite to use an in-memory database since
we do not want the file system interfering with benchmark results.

goos: darwin
goarch: arm64
pkg: github.com/mattn/go-sqlite3
cpu: Apple M1 Max
                             │ base.10.txt  │             new.10.txt              │
                             │    sec/op    │   sec/op     vs base                │
CustomFunctions-10              3.318µ ± 2%   3.115µ ± 2%   -6.10% (p=0.000 n=10)
Suite/BenchmarkExec-10          1.236µ ± 1%   1.240µ ± 2%        ~ (p=0.617 n=10)
Suite/BenchmarkQuery-10         4.004µ ± 7%   3.363µ ± 2%  -16.02% (p=0.000 n=10)
Suite/BenchmarkParams-10        4.241µ ± 1%   3.758µ ± 2%  -11.40% (p=0.000 n=10)
Suite/BenchmarkStmt-10          2.830µ ± 0%   2.378µ ± 2%  -15.97% (p=0.000 n=10)
Suite/BenchmarkRows-10          126.3µ ± 1%   101.3µ ± 1%  -19.79% (p=0.000 n=10)
Suite/BenchmarkStmtRows-10      124.9µ ± 1%   100.5µ ± 2%  -19.56% (p=0.000 n=10)
Suite/BenchmarkStmt10Cols-10   10.130µ ± 0%   7.042µ ± 1%  -30.48% (p=0.000 n=10)
geomean                         8.655µ        7.328µ       -15.33%

                             │ base.10.txt  │               new.10.txt               │
                             │     B/op     │     B/op      vs base                  │
CustomFunctions-10               568.0 ± 0%     576.0 ± 0%   +1.41% (p=0.000 n=10)
Suite/BenchmarkExec-10           128.0 ± 0%     128.0 ± 0%        ~ (p=1.000 n=10) ¹
Suite/BenchmarkQuery-10          688.0 ± 0%     648.0 ± 0%   -5.81% (p=0.000 n=10)
Suite/BenchmarkParams-10       1.078Ki ± 0%   1.031Ki ± 0%   -4.35% (p=0.000 n=10)
Suite/BenchmarkStmt-10           920.0 ± 0%     872.0 ± 0%   -5.22% (p=0.000 n=10)
Suite/BenchmarkRows-10         9.305Ki ± 0%   9.188Ki ± 0%   -1.26% (p=0.000 n=10)
Suite/BenchmarkStmtRows-10     9.289Ki ± 0%   9.164Ki ± 0%   -1.35% (p=0.000 n=10)
Suite/BenchmarkStmt10Cols-10     992.0 ± 0%     696.0 ± 0%  -29.84% (p=0.000 n=10)
geomean                        1.181Ki        1.106Ki        -6.35%
¹ all samples are equal

                             │ base.10.txt │              new.10.txt              │
                             │  allocs/op  │ allocs/op   vs base                  │
CustomFunctions-10              18.00 ± 0%   18.00 ± 0%        ~ (p=1.000 n=10) ¹
Suite/BenchmarkExec-10          7.000 ± 0%   7.000 ± 0%        ~ (p=1.000 n=10) ¹
Suite/BenchmarkQuery-10         23.00 ± 0%   23.00 ± 0%        ~ (p=1.000 n=10) ¹
Suite/BenchmarkParams-10        27.00 ± 0%   27.00 ± 0%        ~ (p=1.000 n=10) ¹
Suite/BenchmarkStmt-10          25.00 ± 0%   25.00 ± 0%        ~ (p=1.000 n=10) ¹
Suite/BenchmarkRows-10          525.0 ± 0%   519.0 ± 0%   -1.14% (p=0.000 n=10)
Suite/BenchmarkStmtRows-10      524.0 ± 0%   518.0 ± 0%   -1.15% (p=0.000 n=10)
Suite/BenchmarkStmt10Cols-10    39.00 ± 0%   19.00 ± 0%  -51.28% (p=0.000 n=10)
geomean                         46.26        42.17        -8.86%
¹ all samples are equal
    @charlievieth
    charlievieth committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    8cf69a9 View commit details
    Browse the repository at this point in the history