Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

658 snippet support sqlcmd #721

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
a40bd59
snippet support for profile and explore
AnirudhVIyer Jul 11, 2023
120ccdb
snippet support for profile and explore 1
AnirudhVIyer Jul 11, 2023
f2deca6
Update table_explorer.ipynb
AnirudhVIyer Jul 11, 2023
e9251f7
Merge branch 'master' into 658-snippet-support-sqlcmd
AnirudhVIyer Jul 12, 2023
fc11cc4
review changes 2
AnirudhVIyer Jul 12, 2023
ddb7cbf
Update table_explorer.ipynb
AnirudhVIyer Jul 12, 2023
44bdcda
fix lint
AnirudhVIyer Jul 12, 2023
62c5b83
Merge branch '658-snippet-support-sqlcmd' of https://github.com/Aniru…
AnirudhVIyer Jul 12, 2023
023d366
fix errors
AnirudhVIyer Jul 12, 2023
901a2c4
fix errors review
AnirudhVIyer Jul 12, 2023
e9aabba
modyfying snippet message
AnirudhVIyer Jul 13, 2023
aea58f5
refactored is_snippet and with-clause
AnirudhVIyer Jul 14, 2023
ed2ff7c
refactored is_snippet and with-clause removed
AnirudhVIyer Jul 14, 2023
466178d
Merge branch 'ploomber:master' into 658-snippet-support-sqlcmd
AnirudhVIyer Jul 14, 2023
7276af3
is_snippet refactor, remove with_clause
AnirudhVIyer Jul 14, 2023
49ca7a7
Merge branch '658-snippet-support-sqlcmd' of https://github.com/Aniru…
AnirudhVIyer Jul 14, 2023
2f53599
Empty-Commit
AnirudhVIyer Jul 14, 2023
44652df
use dictionary to select messaged
AnirudhVIyer Jul 17, 2023
2651854
use dictionary to select messages
AnirudhVIyer Jul 17, 2023
9322172
use dictionary to select messages 1
AnirudhVIyer Jul 17, 2023
2d34975
Merge branch 'master' into 658-snippet-support-sqlcmd
AnirudhVIyer Jul 17, 2023
a677d8b
use dictionary to select messages 2
AnirudhVIyer Jul 17, 2023
0b2069d
Update duckdb-native-sqlalchemy.md
AnirudhVIyer Jul 17, 2023
7a6ec04
use dictionary to select messages 3
AnirudhVIyer Jul 17, 2023
6920d50
replace store_query with prep_query
AnirudhVIyer Jul 18, 2023
0c235df
replace store_query with prep_query 1
AnirudhVIyer Jul 18, 2023
40a63e9
Merge branch 'master' into 658-snippet-support-sqlcmd
AnirudhVIyer Jul 18, 2023
8abe684
Update duckdb-native-sqlalchemy.md
AnirudhVIyer Jul 18, 2023
f4e7e86
replace store_query with prep_query 2
AnirudhVIyer Jul 18, 2023
ae843a6
replace store_query with prep_query 3
AnirudhVIyer Jul 18, 2023
040f9eb
replace store_query with prep_query 4
AnirudhVIyer Jul 18, 2023
6d6b6d3
replace store_query with prep_query 5
AnirudhVIyer Jul 18, 2023
f391844
restore store_render
AnirudhVIyer Jul 19, 2023
5d3fbbb
restore prepare_query
AnirudhVIyer Jul 19, 2023
f612d15
Merge branch 'master' into 658-snippet-support-sqlcmd
AnirudhVIyer Jul 20, 2023
839224e
integration test added
AnirudhVIyer Jul 20, 2023
8919b22
Merge branch 'master' into 658-snippet-support-sqlcmd
AnirudhVIyer Jul 27, 2023
4f98ffd
refactor changes
AnirudhVIyer Jul 27, 2023
1e044ac
resolve merge conflicts
AnirudhVIyer Jul 27, 2023
46189c0
resolve merge conflicts 1
AnirudhVIyer Jul 27, 2023
b05c54e
resolve merge conflicts 2
AnirudhVIyer Jul 27, 2023
9fe8de6
resolve merge conflicts 3
AnirudhVIyer Jul 27, 2023
8507eca
Merge branch 'master' into 658-snippet-support-sqlcmd
AnirudhVIyer Jul 27, 2023
056c876
resolve merge conflicts 3
AnirudhVIyer Jul 27, 2023
04b77a3
merge conflict 4
AnirudhVIyer Jul 27, 2023
213757f
changes
AnirudhVIyer Jul 31, 2023
0194b32
Merge branch 'master' into 658-snippet-support-sqlcmd
AnirudhVIyer Aug 4, 2023
f0feedb
update tests
AnirudhVIyer Aug 7, 2023
e3d2889
update tests
AnirudhVIyer Aug 7, 2023
b28fed3
test commit
AnirudhVIyer Aug 8, 2023
e1049c1
change integration test
AnirudhVIyer Aug 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
* [Doc] Add Redshift tutorial
* [Feature] Adds Redshift support for `%sqlplot boxplot`
* [Fix] Improves performance when converting DuckDB results to `pandas.DataFrame`
* [Fix] Added support for `profile` and `explore` commands with saved snippets (#658)

## 0.9.0 (2023-08-01)

Expand Down Expand Up @@ -41,6 +42,7 @@
* [Doc] Document `--persist-replace` in API section ([#539](https://github.com/ploomber/jupysql/issues/539))
* [Doc] Re-organized sections. Adds section showing how to share notebooks via Ploomber Cloud


## 0.7.9 (2023-06-19)

* [Feature] Modified `histogram` command to support data with NULL values ([#176](https://github.com/ploomber/jupysql/issues/176))
Expand Down
1 change: 1 addition & 0 deletions doc/tutorials/duckdb-native-sqlalchemy.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ df = pd.DataFrame(np.random.randn(num_rows, num_cols))

```{code-cell} ipython3
import duckdb

conn = duckdb.connect()
```

Expand Down
56 changes: 29 additions & 27 deletions src/sql/inspect.py
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ class Columns(DatabaseInspection):
"""

def __init__(self, name, schema, conn=None) -> None:
util.is_table_exists(name, schema)
util.is_saved_snippet_or_table_exists(name, schema)

inspector = _get_inspector(conn)

Expand Down Expand Up @@ -230,14 +230,19 @@ class TableDescription(DatabaseInspection):
"""

def __init__(self, table_name, schema=None) -> None:
util.is_table_exists(table_name, schema)
with_ = util.is_saved_snippet_or_table_exists(
table_name, schema, task="profile"
)

if schema:
table_name = f"{schema}.{table_name}"

conn = ConnectionManager.current

columns_query_result = conn.raw_execute(f"SELECT * FROM {table_name} WHERE 1=0")
columns_query_result = conn.execute(
f"SELECT * FROM {table_name} WHERE 1=0", with_=with_
)

if ConnectionManager.current.is_dbapi_connection:
columns = [i[0] for i in columns_query_result.description]
else:
Expand All @@ -253,8 +258,8 @@ def __init__(self, table_name, schema=None) -> None:

# check the datatype of a column
try:
result = ConnectionManager.current.raw_execute(
f"""SELECT {column} FROM {table_name} LIMIT 1"""
result = conn.execute(
f"SELECT {column} FROM {table_name} LIMIT 1", with_=with_
).fetchone()

value = result[0]
Expand All @@ -269,10 +274,11 @@ def __init__(self, table_name, schema=None) -> None:
message_check = True
# Note: index is reserved word in sqlite
try:
result_col_freq_values = ConnectionManager.current.raw_execute(
result_col_freq_values = conn.execute(
f"""SELECT DISTINCT {column} as top,
COUNT({column}) as frequency FROM {table_name}
GROUP BY top ORDER BY frequency Desc""",
with_=with_,
).fetchall()

table_stats[column]["freq"] = result_col_freq_values[0][1]
Expand All @@ -285,14 +291,13 @@ def __init__(self, table_name, schema=None) -> None:

try:
# get all non None values, min, max and avg.
result_value_values = ConnectionManager.current.raw_execute(
f"""
SELECT MIN({column}) AS min,
result_value_values = conn.execute(
f"""SELECT MIN({column}) AS min,
MAX({column}) AS max,
COUNT({column}) AS count
FROM {table_name}
WHERE {column} IS NOT NULL
""",
WHERE {column} IS NOT NULL""",
with_=with_,
).fetchall()

columns_to_include_in_report.update(["count", "min", "max"])
Expand All @@ -308,26 +313,24 @@ def __init__(self, table_name, schema=None) -> None:

try:
# get unique values
result_value_values = ConnectionManager.current.raw_execute(
f"""
SELECT
result_value_values = conn.execute(
f""" SELECT
COUNT(DISTINCT {column}) AS unique_count
FROM {table_name}
WHERE {column} IS NOT NULL
""",
WHERE {column} IS NOT NULL""",
with_=with_,
).fetchall()
table_stats[column]["unique"] = result_value_values[0][0]
columns_to_include_in_report.update(["unique"])
except Exception:
pass

try:
results_avg = ConnectionManager.current.raw_execute(
f"""
SELECT AVG({column}) AS avg
FROM {table_name}
WHERE {column} IS NOT NULL
""",
results_avg = conn.execute(
f""" SELECT AVG({column}) AS avg
FROM {table_name}
WHERE {column} IS NOT NULL""",
with_=with_,
).fetchall()

columns_to_include_in_report.update(["mean"])
Expand All @@ -341,18 +344,17 @@ def __init__(self, table_name, schema=None) -> None:

try:
# Note: stddev_pop and PERCENTILE_DISC will work only on DuckDB
result = ConnectionManager.current.raw_execute(
f"""
SELECT
result = conn.execute(
f""" SELECT
stddev_pop({column}) as key_std,
percentile_disc(0.25) WITHIN GROUP
(ORDER BY {column}) as key_25,
percentile_disc(0.50) WITHIN GROUP
(ORDER BY {column}) as key_50,
percentile_disc(0.75) WITHIN GROUP
(ORDER BY {column}) as key_75
FROM {table_name}
""",
FROM {table_name}""",
with_=with_,
).fetchall()

columns_to_include_in_report.update(special_numeric_keys)
Expand Down
11 changes: 1 addition & 10 deletions src/sql/magic_plot.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ def execute(self, line="", cell="", local_ns=None):
if cmd.args.with_:
with_ = cmd.args.with_
else:
with_ = self._check_table_exists(table)
with_ = util.is_saved_snippet_or_table_exists(table, task="plot")

if cmd.args.line[0] in {"box", "boxplot"}:
return plot.boxplot(
Expand Down Expand Up @@ -127,12 +127,3 @@ def execute(self, line="", cell="", local_ns=None):
show_num=cmd.args.show_numbers,
conn=None,
)

@staticmethod
def _check_table_exists(table):
with_ = None
if util.is_saved_snippet(table):
with_ = [table]
else:
util.is_table_exists(table)
return with_
57 changes: 47 additions & 10 deletions src/sql/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -187,8 +187,15 @@ def strip_multiple_chars(string: str, chars: str) -> str:


def is_saved_snippet(table: str) -> bool:
"""
checks if table is part of a snippet
Parameters
----------
table: str
Table name

"""
if table in list(store):
display.message(f"Plotting using saved snippet : {table}")
return True
return False

Expand Down Expand Up @@ -264,7 +271,7 @@ def support_only_sql_alchemy_connection(command):


def fetch_sql_with_pagination(
table, offset, n_rows, sort_column=None, sort_order=None
table, offset, n_rows, sort_column=None, sort_order=None, with_=None
) -> tuple:
"""
Returns next n_rows and columns from table starting at the offset
Expand All @@ -286,19 +293,22 @@ def fetch_sql_with_pagination(

sort_order : 'DESC' or 'ASC', default None
Order list

with_ : str, default None
Name of the snippet used to generate the table
"""
is_table_exists(table)
if with_ is None:
is_table_exists(table)

order_by = "" if not sort_column else f"ORDER BY {sort_column} {sort_order}"
query = f""" SELECT * FROM {table} {order_by}
LIMIT {n_rows} OFFSET {offset}"""
yafimvo marked this conversation as resolved.
Show resolved Hide resolved

query = f"""
SELECT * FROM {table} {order_by}
OFFSET {offset} ROWS FETCH NEXT {n_rows} ROWS ONLY"""

rows = ConnectionManager.current.execute(query).fetchall()
rows = ConnectionManager.current.execute(query, with_=with_).fetchall()

columns = ConnectionManager.current.raw_execute(
f"SELECT * FROM {table} WHERE 1=0"
columns = ConnectionManager.current.execute(
f""" SELECT * FROM {table} WHERE 1=0""",
with_=with_,
).keys()

return rows, columns
Expand Down Expand Up @@ -374,6 +384,33 @@ def show_deprecation_warning():
)


def is_saved_snippet_or_table_exists(table, schema=None, task=None):
"""
Check if the referenced table is a snippet
neelasha23 marked this conversation as resolved.
Show resolved Hide resolved
Parameters
----------
table : str,
name of table

schema : str, default None
Name of the schema

task: str, default None
Command calling this function

Returns
-------
with_ : str, default None
name of the snippet
"""
with_ = None
if is_saved_snippet(table):
with_ = [table]
else:
is_table_exists(table, schema)
return with_


def find_path_from_root(file_name):
"""
Recursively finds an absolute path to file_name starting
Expand Down
18 changes: 13 additions & 5 deletions src/sql/widgets/table_widget/table_widget.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@
from sql.util import (
fetch_sql_with_pagination,
parse_sql_results_to_json,
is_table_exists,
is_saved_snippet_or_table_exists,
)

from sql.widgets import utils
from sql.telemetry import telemetry

Expand All @@ -32,7 +31,7 @@ def __init__(self, table):

self.html = ""

is_table_exists(table)
self.with_ = is_saved_snippet_or_table_exists(table, task="explore")

# load css
html_style = utils.load_css(f"{BASE_DIR}/css/tableWidget.css")
Expand All @@ -57,11 +56,19 @@ def create_table(self, table):
Creates an HTML table with default data
"""
rows_per_page = 10
rows, columns = fetch_sql_with_pagination(table, 0, rows_per_page)
rows, columns = fetch_sql_with_pagination(
table, 0, rows_per_page, with_=self.with_
)
rows = parse_sql_results_to_json(rows, columns)

query = f"SELECT count(*) FROM {table}"
query = str(
ConnectionManager.current._prepare_query(
f""" SELECT count(*) FROM {table}""",
with_=self.with_,
)
)
n_total = ConnectionManager.current.raw_execute(query).fetchone()[0]

table_name = table.strip('"').strip("'")

n_pages = math.ceil(n_total / rows_per_page)
Expand Down Expand Up @@ -184,6 +191,7 @@ def _recv(msg):
n_rows,
sort_column=sort_column,
sort_order=sort_order,
with_=self.with_,
)
rows_json = parse_sql_results_to_json(rows, columns)

Expand Down
48 changes: 47 additions & 1 deletion src/tests/integration/test_generic_db_operations.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from sql.error_message import CTE_MSG
from unittest.mock import ANY, Mock
from IPython.core.error import UsageError

from IPython.utils.io import capture_output
import math

ALL_DATABASES = [
Expand Down Expand Up @@ -1095,3 +1095,49 @@ def test_autocommit_create_table_multiple_cells(
).result

assert len(result) == 3


@pytest.mark.parametrize("ip_with_dynamic_db", ALL_DATABASES)
def test_explore_fetch_sql_with_pagination(
ip_with_dynamic_db,
request,
tmp_empty,
):
if ip_with_dynamic_db not in [
"ip_with_postgreSQL",
"ip_with_duckDB",
"ip_with_SQLite",
]:
pytest.xfail(reason="There may be an issue due to code refactoring")
Comment on lines +1106 to +1111

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let the tests run and only exclude the ones where we are sure they'll fail


ip_with_dynamic_db = request.getfixturevalue(ip_with_dynamic_db)

ip_with_dynamic_db.run_cell(
"""
%%sql
CREATE TABLE people (name varchar(50), number int, country varchar(50));
INSERT INTO people VALUES ('joe', 82, 'usa');
INSERT INTO people VALUES ('paula', 93, 'uk');
INSERT INTO people VALUES ('sam', 77, 'canada');
INSERT INTO people VALUES ('emily', 65, 'usa');
INSERT INTO people VALUES ('michael', 89, 'usa');
INSERT INTO people VALUES ('sarah', 81, 'uk');
INSERT INTO people VALUES ('james', 76, 'usa');
INSERT INTO people VALUES ('angela', 88, 'usa');
INSERT INTO people VALUES ('robert', 82, 'usa');
INSERT INTO people VALUES ('lisa', 92, 'uk');
INSERT INTO people VALUES ('mark', 77, 'usa');
INSERT INTO people VALUES ('jennifer', 68, 'australia');
"""
)
ip_with_dynamic_db.run_cell(
"""
%%sql --save citizen
select * from people where country = 'usa'
"""
)
Comment on lines +1117 to +1138

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're already creating some tables for all databases (although they have different names), check out how we're getting the table names. there's no need to create new tables here

with capture_output() as captured:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're using pytests capsys for this. it works pretty much the same, let's replace it so our tests are consistent

ip_with_dynamic_db.run_cell("%sqlcmd explore -t citizen")

assert "sql.widgets.table_widget.table_widget.TableWidget object" in captured.stdout
ip_with_dynamic_db.run_cell("""%sql DROP TABLE people""")
Loading
Loading