Skip to content

Commit

Permalink
Add SQLite Catalog support (#178)
Browse files Browse the repository at this point in the history
Co-authored-by: Brian "bits" Olsen <[email protected]>
  • Loading branch information
Fokko and bitsondatadev authored Dec 11, 2023
1 parent e92e10a commit 8c8abb5
Show file tree
Hide file tree
Showing 5 changed files with 433 additions and 170 deletions.
21 changes: 19 additions & 2 deletions mkdocs/docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,9 @@ catalog:

## SQL Catalog

The SQL catalog requires a database for its backend. As of now, pyiceberg only supports PostgreSQL through psycopg2.
The database connection has to be configured using the `uri` property (see SQLAlchemy's [documentation for URL format](https://docs.sqlalchemy.org/en/20/core/engines.html#backend-specific-urls)):
The SQL catalog requires a database for its backend. PyIceberg supports PostgreSQL and SQLite through psycopg2. The database connection has to be configured using the `uri` property. See SQLAlchemy's [documentation for URL format](https://docs.sqlalchemy.org/en/20/core/engines.html#backend-specific-urls):

For PostgreSQL:

```yaml
catalog:
Expand All @@ -150,6 +151,22 @@ catalog:
uri: postgresql+psycopg2://username:password@localhost/mydatabase
```

In the case of SQLite:

<!-- prettier-ignore-start -->

!!! warning inline end "Development only"
SQLite is not built for concurrency, you should use this catalog for exploratory or development purposes.

<!-- prettier-ignore-end -->

```yaml
catalog:
default:
type: sql
uri: sqlite:////tmp/pyiceberg.db
```

## Hive Catalog

```yaml
Expand Down
19 changes: 15 additions & 4 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion pyiceberg/catalog/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ def infer_catalog_type(name: str, catalog_properties: RecursiveDict) -> Optional
return CatalogType.REST
elif uri.startswith("thrift"):
return CatalogType.HIVE
elif uri.startswith("postgresql"):
elif uri.startswith(("sqlite", "postgresql")):
return CatalogType.SQL
else:
raise ValueError(f"Could not infer the catalog type from the uri: {uri}")
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ sqlalchemy = { version = "^2.0.18", optional = true }
[tool.poetry.dev-dependencies]
pytest = "7.4.3"
pytest-checkdocs = "2.10.1"
pytest-lazy-fixture = "0.6.3"
pre-commit = "3.5.0"
fastavro = "1.9.1"
coverage = { version = "^7.3.2", extras = ["toml"] }
Expand Down
Loading

0 comments on commit 8c8abb5

Please sign in to comment.