add duckdb support #1398

ahuang11 · 2024-08-22T15:41:17Z

import hvplot.duckdb

import pandas as pd
import duckdb

df = pd.DataFrame({
    'x': [1, 2, 3],
    'y': [4, 5, 6]
})

df.to_parquet('test.parquet')

connection = duckdb.connect(database=':memory:', read_only=False)
connection.from_parquet("test.parquet").hvplot(
    x='x', y='y', kind='scatter'
)

codecov · 2024-08-22T16:01:58Z

Codecov Report

Attention: Patch coverage is 94.79167% with 5 lines in your changes missing coverage. Please review.

Project coverage is 88.94%. Comparing base (efeda78) to head (e008752).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
hvplot/duckdb.py	87.50%	2 Missing ⚠️
hvplot/tests/testpatch.py	89.47%	2 Missing ⚠️
hvplot/plotting/core.py	96.87%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1398      +/-   ##
==========================================
+ Coverage   88.73%   88.94%   +0.20%     
==========================================
  Files          51       52       +1     
  Lines        7592     7751     +159     
==========================================
+ Hits         6737     6894     +157     
- Misses        855      857       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

MarcSkovMadsen · 2024-08-22T16:30:42Z

Is the long term goal to implement a real duckdb backend for more efficient usage?

ahuang11 · 2024-08-22T16:32:25Z

Can you elaborate on real duckdb backend?

This PR doesn't change the converter at all; only patches duckdb so DuckDBPyRelation.hvplot() is available.

It does not do DuckDBPyRelation.fetchdf().hvplot() AFAIK.

MarcSkovMadsen · 2024-08-23T16:06:25Z

Yes.

Is the plan to implement a real HoloViz backend such that you only query the data needed for the plot? For example when using groupby.

ahuang11 · 2024-08-28T17:23:22Z

I'm not sure I follow; this PR allows hvplot off of a DuckDBRelation, which is already the result of a query.

connection > relation (select * from table where ...) > hvplot

connection.execute("SELECT * FROM table WHERE col = 'A'").hvplot()

Ah I see the confusion:

connection.from_parquet("test.parquet").hvplot(
    x='x', y='y', kind='scatter'
)

Can be drilled down further

connection.from_parquet("test.parquet").execute(...).hvplot(
    x='x', y='y', kind='scatter'
)

ahuang11 · 2024-08-28T20:50:57Z

Was slightly mistaken, but a little modification works:
https://github.com/holoviz/hvplot/pull/1398/files#diff-a47979cba5da76fbc1aec07d45b4403241da5cfbbbf6b36652f75c52fb644824R360

Not sure if we should manually add the optimization to subset x/y columns (e.g. relation.select(x, y), or let the user do it themselves.

hvplot/duckdb.py

ahuang11 · 2024-08-28T20:55:31Z

Oh I guess it is converting to pandas object first:
lambda self: hvPlotTabular(self.df())

https://github.com/holoviz/hvplot/pull/1398/files#diff-be8c4d86a0e1601a53aca277191c3a5d8e160fc39c2b4d75396c83b6a38ae610R15

Edit: Now it does what you proposed

hvplot/plotting/__init__.py

maximlt

I'm ok on adding DuckDB support. It looks like you should update the diagram on the landing page too, unless you consider this experimental first.

doc/index.md

doc/user_guide/Integrations.ipynb

maximlt · 2024-09-13T12:36:36Z

@ahuang11 can you also update this PR based on #1359 now it has been merged?

ahuang11 · 2024-09-13T21:23:04Z

Okay I think this is ready. I added the diagrams + added changes from #1359

doc/assets/diagram.svg

doc/index.md

Co-authored-by: Maxime Liquet <[email protected]>

hvplot/plotting/__init__.py

Co-authored-by: Simon Høxbro Hansen <[email protected]>

hvplot/tests/testplotting.py

ahuang11 marked this pull request as ready for review August 28, 2024 20:52

ahuang11 requested a review from hoxbro August 28, 2024 20:53

ahuang11 commented Aug 28, 2024

View reviewed changes

hvplot/duckdb.py Outdated Show resolved Hide resolved

ahuang11 commented Aug 28, 2024

View reviewed changes

hvplot/duckdb.py Outdated Show resolved Hide resolved

ahuang11 commented Aug 28, 2024

View reviewed changes

hvplot/duckdb.py Outdated Show resolved Hide resolved

ahuang11 commented Sep 11, 2024

View reviewed changes

hvplot/plotting/__init__.py Outdated Show resolved Hide resolved

ahuang11 requested a review from maximlt September 11, 2024 09:19

maximlt reviewed Sep 12, 2024

View reviewed changes

doc/index.md Show resolved Hide resolved

doc/user_guide/Integrations.ipynb Show resolved Hide resolved

doc/user_guide/Integrations.ipynb Outdated Show resolved Hide resolved

maximlt added this to the 0.11.0 milestone Sep 13, 2024

ahuang11 and others added 5 commits September 13, 2024 14:13

add duckdb support

90a2702

add test

ceb4805

add doc

d9d54f2

Update hvplot/duckdb.py

5706cbe

Apply suggestions from code review

1bb8b44

ahuang11 force-pushed the add_duckdb branch from 01fe825 to 6abcb7d Compare September 13, 2024 21:13

ahuang11 added 7 commits September 13, 2024 14:14

improve support for duckdb

72b9317

fix duckdb cononection and tests

a9bf8e2

fix test

c02d6de

fix test

ebac847

fix docs

6abcb7d

update diagram

5fc2da4

up size

498aa23

ahuang11 requested a review from maximlt September 13, 2024 21:19

svg size

06e6500

rebase on the other pr

9ed44f2

maximlt reviewed Sep 20, 2024

View reviewed changes

doc/assets/diagram.svg Outdated Show resolved Hide resolved

maximlt reviewed Sep 20, 2024

View reviewed changes

doc/index.md Outdated Show resolved Hide resolved

ahuang11 and others added 2 commits September 23, 2024 07:43

update diagram

ae74adc

Update doc/index.md

0c6efdc

Co-authored-by: Maxime Liquet <[email protected]>

ahuang11 requested a review from maximlt September 23, 2024 14:44

philippjfr approved these changes Sep 24, 2024

View reviewed changes

hoxbro reviewed Sep 24, 2024

View reviewed changes

hvplot/plotting/__init__.py Outdated Show resolved Hide resolved

Update hvplot/plotting/__init__.py

ff9aed1

Co-authored-by: Simon Høxbro Hansen <[email protected]>

hoxbro reviewed Sep 25, 2024

View reviewed changes

hvplot/tests/testplotting.py Outdated Show resolved Hide resolved

remove redondant test

e008752

maximlt approved these changes Sep 25, 2024

View reviewed changes

maximlt merged commit c689c60 into main Sep 25, 2024
9 checks passed

maximlt deleted the add_duckdb branch September 25, 2024 16:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add duckdb support #1398

add duckdb support #1398

ahuang11 commented Aug 22, 2024 •

edited

Loading

codecov bot commented Aug 22, 2024 •

edited

Loading

MarcSkovMadsen commented Aug 22, 2024

ahuang11 commented Aug 22, 2024 •

edited

Loading

MarcSkovMadsen commented Aug 23, 2024

ahuang11 commented Aug 28, 2024 •

edited

Loading

ahuang11 commented Aug 28, 2024 •

edited

Loading

ahuang11 commented Aug 28, 2024 •

edited

Loading

maximlt left a comment

maximlt commented Sep 13, 2024

ahuang11 commented Sep 13, 2024

add duckdb support #1398

add duckdb support #1398

Conversation

ahuang11 commented Aug 22, 2024 • edited Loading

codecov bot commented Aug 22, 2024 • edited Loading

Codecov Report

MarcSkovMadsen commented Aug 22, 2024

ahuang11 commented Aug 22, 2024 • edited Loading

MarcSkovMadsen commented Aug 23, 2024

ahuang11 commented Aug 28, 2024 • edited Loading

ahuang11 commented Aug 28, 2024 • edited Loading

ahuang11 commented Aug 28, 2024 • edited Loading

maximlt left a comment

Choose a reason for hiding this comment

maximlt commented Sep 13, 2024

ahuang11 commented Sep 13, 2024

ahuang11 commented Aug 22, 2024 •

edited

Loading

codecov bot commented Aug 22, 2024 •

edited

Loading

ahuang11 commented Aug 22, 2024 •

edited

Loading

ahuang11 commented Aug 28, 2024 •

edited

Loading

ahuang11 commented Aug 28, 2024 •

edited

Loading

ahuang11 commented Aug 28, 2024 •

edited

Loading