Skip to content

Commit

Permalink
update zstd_vfs with VFS logging
Browse files Browse the repository at this point in the history
  • Loading branch information
mlin committed Feb 28, 2023
1 parent d248a12 commit efaefe5
Show file tree
Hide file tree
Showing 3 changed files with 7 additions and 5 deletions.
2 changes: 1 addition & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ endif()
FetchContent_Declare(
sqlite_zstd_vfs
GIT_REPOSITORY https://github.com/mlin/sqlite_zstd_vfs.git
GIT_TAG 25e962e
GIT_TAG 2e9c7db
)
FetchContent_MakeAvailable(sqlite_zstd_vfs)
FetchContent_MakeAvailable(sqlitecpp)
Expand Down
2 changes: 1 addition & 1 deletion docs/guide_db.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ The **GenomicSQLite Open** routine and the `genomicsqlite` shell also accept htt
Under the hood, the extension uses [libcurl](https://curl.se/libcurl/) to send web requests for necessary portions of the database file as queries proceed, with adaptive batching & prefetching to balance the number and size of these requests. This works well for point lookups and queries that scan largely-contiguous slices of tables and indexes (and a modest number thereof). It's less suitable for big multi-way joins and other aggressively random access patterns; in such cases, it'd be better to download the database file upfront to open locally.

* Reading large databases over the web, budget an additional ~600MiB of memory for HTTP prefetch buffers.
* The HTTP driver writes log messages to standard error when requests fail or had to be retried, which can be disabled by setting configuration web_log = 0 or environment SQLITE_WEB_LOG=0; or increased up to 5 to log every request and other details.
* The HTTP driver writes log messages to standard error when requests fail or had to be retried, which can be disabled by setting configuration vfs_log = 0 or environment SQLITE_VFS_LOG=0; or increased up to 5 for extensive debug logging.
* To disable TLS certificate and hostname verification, set web_insecure = true in the GenomicSQLite configuration, or SQLITE_WEB_INSECURE=1 in the environment.
* The above-described `genomicsqlite DB_FILENAME --compact` optimizes a database for web access by making the request pattern more contiguous.

Expand Down
8 changes: 5 additions & 3 deletions src/genomicsqlite.cc
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ std::string GenomicSQLiteDefaultConfigJSON() {
"zstd_level": 6,
"inner_page_KiB": 16,
"outer_page_KiB": 32,
"web_log": 2,
"vfs_log": -1,
"web_insecure": false,
"web_dbi_url": "",
"web_nodbi": false,
Expand Down Expand Up @@ -173,9 +173,11 @@ string GenomicSQLiteURI(const string &dbfile, const string &config_json = "") {
bool web = dbfile.substr(0, 5) == "http:" || dbfile.substr(0, 6) == "https:";
ostringstream uri;
uri << "file:" << (web ? "/__web__" : SQLiteNested::urlencode(dbfile, true)) << "?vfs=zstd";
if (cfg.GetInt("$.vfs_log") >= 0) {
uri << "&vfs_log=" << cfg.GetInt("$.vfs_log");
}
if (web) {
uri << "&mode=ro&immutable=1&web_url=" << SQLiteNested::urlencode(dbfile)
<< "&web_log=" << cfg.GetInt("$.web_log");
uri << "&mode=ro&immutable=1&web_url=" << SQLiteNested::urlencode(dbfile);
if (cfg.GetBool("$.web_insecure")) {
uri << "&web_insecure=1";
}
Expand Down

0 comments on commit efaefe5

Please sign in to comment.