Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SNOW-1830018: Snowpark not able to handle python None insertion to snowflake tables #2685

Closed
uthusoo opened this issue Nov 26, 2024 · 2 comments
Closed

SNOW-1830018: Snowpark not able to handle python None insertion to snowflake tables #2685

uthusoo opened this issue Nov 26, 2024 · 2 comments
Assignees
@sfc-gh-sghosh
Labels
status-triage_done Initial triage done, will be further handled by the driver team

Comments

@uthusoo
Copy link

uthusoo commented Nov 26, 2024

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using? Python 3.11.9

  2. What operating system and processor architecture are you using? Windows-10-10.0.19045-SP0

  3. What are the component versions in the environment (pip freeze)?

snowflake-connector-python==3.12.3
snowflake-snowpark-python==1.25.0
snowflake-sqlalchemy==1.5.3

  1. What did you do?
    ############ RUN DDL IN SNOWFLAKE FOR TEST TABLE ##############

create or replace TABLE COVANTA_STAGING.GROCERY_LIST_TEST (

LIST_ID NUMBER(26,0) NOT NULL,

REF_LIST_ID NUMBER(26,0)

)

COMMENT='Testing Snowpark Null Insert into snowflake'

;

###############################################################
from snowflake.snowpark.session import Session
import pandas as pd

connection_parameters = {
"account": "<account_name>",
"user": "",
"password": "",
"schema": "",
"role": "",
}
session = Session.builder.configs(connection_parameters).create()

data = pd.DataFrame([[1, None], [2, 2], [3, 3]], columns=["LIST_ID", "REF_LIST_ID"])
data_details = data.to_dict("split")
snowpark_load_table_df = session.create_dataframe(data_details["data"], schema=data_details["columns"])

table_name = "GROCERY_LIST_TEST"
mode = "truncate"
column_order = "name"

snowpark_load_table_df.write.save_as_table(
table_name, mode=mode, column_order=column_order
)

  1. What did you expect to see?
    I expected that the DataFrame would be inserted into snowflake, and the Python None would get converted to snowflake NULL.

just like if I were to run simply the below command in snowflake -
insert into COVANTA_STAGING.GROCERY_LIST_TEST values (1, NULL), (2, 2), (3, 3).

Instead it gave the error -

raise error_class(
snowflake.snowpark.exceptions.SnowparkSQLException: (1304): 01b8a28f-0412-4919-000c-0583ddde79d2: 100046 (22003): Number out of representable range: type FIXEDSB16{nullable}, value nan

I have validated that the above insert query on the table and it works and gives result

  1. Can you set logging to DEBUG and collect the logs?

2024-11-26 13:57:12,922 - MainThread connection.py:414 - init() - INFO - Snowflake Connector for Python Version: 3.12.3, Python Version: 3.11.9, Platform: Windows-10-10.0.19045-SP0
2024-11-26 13:57:12,922 - MainThread connection.py:731 - connect() - DEBUG - connect
2024-11-26 13:57:12,923 - MainThread connection.py:1115 - __config() - DEBUG - __config
2024-11-26 13:57:12,923 - MainThread connection.py:1197 - __config() - INFO - Connecting to GLOBAL Snowflake domain
2024-11-26 13:57:12,923 - MainThread connection.py:1278 - __config() - INFO - This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity.
2024-11-26 13:57:12,923 - MainThread converter.py:159 - init() - DEBUG - use_numpy: False
2024-11-26 13:57:12,923 - MainThread converter_issue23517.py:27 - init() - DEBUG - initialized
2024-11-26 13:57:12,924 - MainThread connection.py:942 - __open_connection() - DEBUG - REST API object was created:
2024-11-26 13:57:12,924 - MainThread _auth.py:186 - authenticate() - DEBUG - authenticate
2024-11-26 13:57:12,924 - MainThread _auth.py:222 - authenticate() - DEBUG - account=, user=, database=, schema=, warehouse=, role=, request_id=<request_id>
2024-11-26 13:57:12,924 - MainThread _auth.py:255 - authenticate() - DEBUG - body['data']: {'CLIENT_APP_ID': 'PythonSnowpark', 'CLIENT_APP_VERSION': '1.25.0', 'SVN_REVISION': None, 'ACCOUNT_NAME': 'account name', 'LOGIN_NAME': 'login name', 'CLIENT_ENVIRONMENT': {'APPLICATION': 'PythonSnowpark', 'OS': 'Windows', 'OS_VERSION': 'Windows-10-10.0.19045-SP0', 'PYTHON_VERSION': '3.11.9', 'PYTHON_RUNTIME': 'CPython', 'PYTHON_COMPILER': 'MSC v.1938 64 bit (AMD64)', 'OCSP_MODE': 'FAIL_OPEN', 'TRACING': 10, 'LOGIN_TIMEOUT': None, 'NETWORK_TIMEOUT': None, 'SOCKET_TIMEOUT': None}, 'PASSWORD': '', 'SESSION_PARAMETERS': {'CLIENT_PREFETCH_THREADS': 4}}
2024-11-26 13:57:12,925 - MainThread retry.py:351 - from_int() - DEBUG - Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
2024-11-26 13:57:12,925 - MainThread retry.py:351 - from_int() - DEBUG - Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
2024-11-26 13:57:12,925 - MainThread network.py:1213 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 1/1 active sessions
2024-11-26 13:57:12,925 - MainThread network.py:887 - _request_exec_wrapper() - DEBUG - remaining request timeout: N/A ms, retry cnt: 1
2024-11-26 13:57:12,926 - MainThread network.py:869 - add_request_guid() - DEBUG - Request guid: 052638e5-8eed-4add-8415-05ad840bd56c
2024-11-26 13:57:12,926 - MainThread network.py:1059 - _request_exec() - DEBUG - socket timeout: 60
2024-11-26 13:57:12,928 - MainThread connectionpool.py:1019 - _new_conn() - DEBUG - Starting new HTTPS connection (1): account name:port
2024-11-26 13:57:13,331 - MainThread ssl_wrap_socket.py:79 - ssl_wrap_socket_with_ocsp() - DEBUG - OCSP Mode: FAIL_OPEN, OCSP response cache file name: None
2024-11-26 13:57:13,362 - MainThread ocsp_snowflake.py:491 - reset_cache_dir() - DEBUG - cache directory:
2024-11-26 13:57:13,364 - MainThread ocsp_snowflake.py:529 - reset_ocsp_response_cache_uri() - DEBUG - ocsp_response_cache_uri:
2024-11-26 13:57:13,364 - MainThread ocsp_snowflake.py:532 - reset_ocsp_response_cache_uri() - DEBUG - OCSP_VALIDATION_CACHE size: 293
2024-11-26 13:57:13,364 - MainThread ocsp_snowflake.py:328 - reset_ocsp_dynamic_cache_server_url() - DEBUG - OCSP response cache server is enabled:
2024-11-26 13:57:13,365 - MainThread ocsp_snowflake.py:341 - reset_ocsp_dynamic_cache_server_url() - DEBUG - OCSP dynamic cache server RETRY URL: None
2024-11-26 13:57:13,365 - MainThread ocsp_snowflake.py:970 - validate() - DEBUG - validating certificate: account name
2024-11-26 13:57:13,365 - MainThread ocsp_asn1crypto.py:385 - extract_certificate_chain() - DEBUG - # of certificates: 4
2024-11-26 13:57:13,407 - MainThread ocsp_snowflake.py:1027 - _validate() - DEBUG - ok
2024-11-26 13:57:13,642 - MainThread network.py:1086 - _request_exec() - DEBUG - SUCCESS
2024-11-26 13:57:13,643 - MainThread network.py:1218 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 0/1 active sessions
2024-11-26 13:57:13,644 - MainThread network.py:751 - _post_request() - DEBUG - ret[code] = None, after post request
2024-11-26 13:57:13,644 - MainThread _auth.py:385 - authenticate() - DEBUG - completed authentication
2024-11-26 13:57:13,647 - MainThread connection.py:861 - cursor() - DEBUG - cursor
2024-11-26 13:57:13,648 - MainThread session.py:629 - init() - INFO - Snowpark Session information:
"version" : 1.25.0,
"python.version" : 3.11.9,
"python.connector.version" : 3.12.3,
"python.connector.session.id" : 3383760240866770,
"os.name" : Windows

2024-11-26 13:57:13,652 - MainThread cursor.py:921 - execute() - DEBUG - executing SQL/command
2024-11-26 13:57:13,652 - MainThread cursor.py:936 - execute() - DEBUG - query: [SELECT $1 AS "LIST_ID", $2 AS "REF_LIST_ID" FROM VALUES (0 :: BIGINT, 0 :: FLOA...]
2024-11-26 13:57:13,652 - MainThread connection.py:1654 - _next_sequence_counter() - DEBUG - sequence counter: 1
2024-11-26 13:57:13,653 - MainThread cursor.py:644 - _execute_helper() - DEBUG - Request id: 4f0cca92-d7af-4993-a20a-8935f4af3f76
2024-11-26 13:57:13,653 - MainThread cursor.py:646 - _execute_helper() - DEBUG - running query [SELECT $1 AS "LIST_ID", $2 AS "REF_LIST_ID" FROM VALUES (0 :: BIGINT, 0 :: FLOA...]
2024-11-26 13:57:13,653 - MainThread cursor.py:653 - _execute_helper() - DEBUG - is_file_transfer: True
2024-11-26 13:57:13,654 - MainThread connection.py:1313 - cmd_query() - DEBUG - _cmd_query
2024-11-26 13:57:13,654 - MainThread _query_context_cache.py:155 - serialize_to_dict() - DEBUG - serialize_to_dict() called
2024-11-26 13:57:13,654 - MainThread connection.py:1342 - cmd_query() - DEBUG - sql=[SELECT $1 AS "LIST_ID", $2 AS "REF_LIST_ID" FROM VALUES (0 :: BIGINT, 0 :: FLOA...], sequence_id=[1], is_file_transfer=[False]
2024-11-26 13:57:13,656 - MainThread network.py:488 - request() - DEBUG - Opentelemtry otel injection failed because of: No module named 'opentelemetry'
2024-11-26 13:57:13,657 - MainThread network.py:1213 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name snowflake.com', SessionPool 1/1 active sessions
2024-11-26 13:57:13,657 - MainThread network.py:887 - _request_exec_wrapper() - DEBUG - remaining request timeout: N/A ms, retry cnt: 1
2024-11-26 13:57:13,658 - MainThread network.py:869 - add_request_guid() - DEBUG - Request guid: af3896e3-0fcf-4eec-87e7-235de1e0e47b
2024-11-26 13:57:13,658 - MainThread network.py:1059 - _request_exec() - DEBUG - socket timeout: 60
2024-11-26 13:57:13,709 - MainThread connectionpool.py:474 - _make_request() - DEBUG - https://account name snowflake.com:443 "POST /queries/v1/query-request?requestId=4f0cca92-d7af-4993-a20a-8935f4af3f76&request_guid=af3896e3-0fcf-4eec-87e7-235de1e0e47b HTTP/1.1" 200 None
2024-11-26 13:57:13,711 - MainThread network.py:1086 - _request_exec() - DEBUG - SUCCESS
2024-11-26 13:57:13,711 - MainThread network.py:1218 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name snowflake.com', SessionPool 0/1 active sessions
2024-11-26 13:57:13,711 - MainThread network.py:751 - _post_request() - DEBUG - ret[code] = None, after post request
2024-11-26 13:57:13,711 - MainThread network.py:777 - _post_request() - DEBUG - Query id: 01b8a291-0412-3d32-000c-0583ddde6e62
2024-11-26 13:57:13,712 - MainThread _query_context_cache.py:191 - deserialize_json_dict() - DEBUG - deserialize_json_dict() called: data from server: {'entries': [{'id': 0, 'timestamp': 1732647433723486, 'priority': 0}]}
2024-11-26 13:57:13,712 - MainThread _query_context_cache.py:232 - deserialize_json_dict() - DEBUG - deserialize {'id': 0, 'timestamp': 1732647433723486, 'priority': 0}
2024-11-26 13:57:13,712 - MainThread _query_context_cache.py:101 - _sync_priority_map() - DEBUG - sync_priority_map called priority_map size = 0, new_priority_map size = 1
2024-11-26 13:57:13,712 - MainThread _query_context_cache.py:127 - trim_cache() - DEBUG - trim_cache() called. treeSet size is 1 and cache capacity is 5
2024-11-26 13:57:13,712 - MainThread _query_context_cache.py:136 - trim_cache() - DEBUG - trim_cache() returns. treeSet size is 1 and cache capacity is 5
2024-11-26 13:57:13,712 - MainThread _query_context_cache.py:271 - deserialize_json_dict() - DEBUG - deserialize_json_dict() returns
2024-11-26 13:57:13,713 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647433723486, 0)
2024-11-26 13:57:13,713 - MainThread cursor.py:995 - execute() - DEBUG - sfqid: 01b8a291-0412-3d32-000c-0583ddde6e62
2024-11-26 13:57:13,713 - MainThread cursor.py:1001 - execute() - DEBUG - query execution done
2024-11-26 13:57:13,713 - MainThread cursor.py:1015 - execute() - DEBUG - SUCCESS
2024-11-26 13:57:13,713 - MainThread cursor.py:1034 - execute() - DEBUG - PUT OR GET: False
2024-11-26 13:57:13,714 - MainThread cursor.py:1152 - _init_result_and_meta() - DEBUG - Query result format: arrow
2024-11-26 13:57:13,714 - MainThread cursor.py:1166 - _init_result_and_meta() - INFO - Number of results in first chunk: 0
2024-11-26 13:57:13,715 - MainThread cursor.py:921 - execute() - DEBUG - executing SQL/command
2024-11-26 13:57:13,715 - MainThread cursor.py:936 - execute() - DEBUG - query: [SELECT "LIST_ID", "REF_LIST_ID" FROM ( SELECT $1 AS "LIST_ID", $2 AS "REF_LIST_I...]
2024-11-26 13:57:13,715 - MainThread connection.py:1654 - _next_sequence_counter() - DEBUG - sequence counter: 2
2024-11-26 13:57:13,716 - MainThread cursor.py:644 - _execute_helper() - DEBUG - Request id: ca4e8d90-32fa-4794-ac64-c4567f2b5dcb
2024-11-26 13:57:13,716 - MainThread cursor.py:646 - _execute_helper() - DEBUG - running query [SELECT "LIST_ID", "REF_LIST_ID" FROM ( SELECT $1 AS "LIST_ID", $2 AS "REF_LIST_I...]
2024-11-26 13:57:13,716 - MainThread cursor.py:653 - _execute_helper() - DEBUG - is_file_transfer: True
2024-11-26 13:57:13,717 - MainThread connection.py:1313 - cmd_query() - DEBUG - _cmd_query
2024-11-26 13:57:13,718 - MainThread _query_context_cache.py:155 - serialize_to_dict() - DEBUG - serialize_to_dict() called
2024-11-26 13:57:13,718 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647433723486, 0)
2024-11-26 13:57:13,719 - MainThread _query_context_cache.py:180 - serialize_to_dict() - DEBUG - serialize_to_dict(): data to send to server {'entries': [{'id': 0, 'timestamp': 1732647433723486, 'priority': 0, 'context': {}}]}
2024-11-26 13:57:13,719 - MainThread connection.py:1342 - cmd_query() - DEBUG - sql=[SELECT "LIST_ID", "REF_LIST_ID" FROM ( SELECT $1 AS "LIST_ID", $2 AS "REF_LIST_I...], sequence_id=[2], is_file_transfer=[False]
2024-11-26 13:57:13,721 - MainThread network.py:488 - request() - DEBUG - Opentelemtry otel injection failed because of: No module named 'opentelemetry'
2024-11-26 13:57:13,722 - MainThread network.py:1213 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 1/1 active sessions
2024-11-26 13:57:13,722 - MainThread network.py:887 - _request_exec_wrapper() - DEBUG - remaining request timeout: N/A ms, retry cnt: 1
2024-11-26 13:57:13,723 - MainThread network.py:869 - add_request_guid() - DEBUG - Request guid: 826a6ea4-708e-4e35-b6d8-5ea4a752c2c0
2024-11-26 13:57:13,723 - MainThread network.py:1059 - _request_exec() - DEBUG - socket timeout: 60
2024-11-26 13:57:13,766 - MainThread connectionpool.py:474 - _make_request() - DEBUG - https://account name:443 "POST /queries/v1/query-request?requestId=ca4e8d90-32fa-4794-ac64-c4567f2b5dcb&request_guid=826a6ea4-708e-4e35-b6d8-5ea4a752c2c0 HTTP/1.1" 200 None
2024-11-26 13:57:13,767 - MainThread network.py:1086 - _request_exec() - DEBUG - SUCCESS
2024-11-26 13:57:13,768 - MainThread network.py:1218 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 0/1 active sessions
2024-11-26 13:57:13,769 - MainThread network.py:751 - _post_request() - DEBUG - ret[code] = None, after post request
2024-11-26 13:57:13,769 - MainThread network.py:777 - _post_request() - DEBUG - Query id: 01b8a291-0412-4919-000c-0583ddde7d9e
2024-11-26 13:57:13,769 - MainThread _query_context_cache.py:191 - deserialize_json_dict() - DEBUG - deserialize_json_dict() called: data from server: {'entries': [{'id': 0, 'timestamp': 1732647433790899, 'priority': 0}]}
2024-11-26 13:57:13,770 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647433723486, 0)
2024-11-26 13:57:13,770 - MainThread _query_context_cache.py:232 - deserialize_json_dict() - DEBUG - deserialize {'id': 0, 'timestamp': 1732647433790899, 'priority': 0}
2024-11-26 13:57:13,771 - MainThread _query_context_cache.py:101 - _sync_priority_map() - DEBUG - sync_priority_map called priority_map size = 0, new_priority_map size = 1
2024-11-26 13:57:13,771 - MainThread _query_context_cache.py:127 - trim_cache() - DEBUG - trim_cache() called. treeSet size is 1 and cache capacity is 5
2024-11-26 13:57:13,771 - MainThread _query_context_cache.py:136 - trim_cache() - DEBUG - trim_cache() returns. treeSet size is 1 and cache capacity is 5
2024-11-26 13:57:13,772 - MainThread _query_context_cache.py:271 - deserialize_json_dict() - DEBUG - deserialize_json_dict() returns
2024-11-26 13:57:13,772 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647433790899, 0)
2024-11-26 13:57:13,773 - MainThread cursor.py:995 - execute() - DEBUG - sfqid: 01b8a291-0412-4919-000c-0583ddde7d9e
2024-11-26 13:57:13,773 - MainThread cursor.py:1001 - execute() - DEBUG - query execution done
2024-11-26 13:57:13,773 - MainThread cursor.py:1015 - execute() - DEBUG - SUCCESS
2024-11-26 13:57:13,774 - MainThread cursor.py:1034 - execute() - DEBUG - PUT OR GET: False
2024-11-26 13:57:13,774 - MainThread cursor.py:1152 - _init_result_and_meta() - DEBUG - Query result format: arrow
2024-11-26 13:57:13,774 - MainThread cursor.py:1166 - _init_result_and_meta() - INFO - Number of results in first chunk: 0
2024-11-26 13:57:13,775 - MainThread cursor.py:921 - execute() - DEBUG - executing SQL/command
2024-11-26 13:57:13,775 - MainThread cursor.py:936 - execute() - DEBUG - query: [show tables like 'GROCERY_LIST_TEST']
2024-11-26 13:57:13,775 - MainThread connection.py:1654 - _next_sequence_counter() - DEBUG - sequence counter: 3
2024-11-26 13:57:13,775 - MainThread cursor.py:644 - _execute_helper() - DEBUG - Request id: 02a977e4-7776-4420-9a23-16cd05a46818
2024-11-26 13:57:13,775 - MainThread cursor.py:646 - _execute_helper() - DEBUG - running query [show tables like 'GROCERY_LIST_TEST']
2024-11-26 13:57:13,776 - MainThread cursor.py:653 - _execute_helper() - DEBUG - is_file_transfer: True
2024-11-26 13:57:13,776 - MainThread connection.py:1313 - cmd_query() - DEBUG - _cmd_query
2024-11-26 13:57:13,776 - MainThread _query_context_cache.py:155 - serialize_to_dict() - DEBUG - serialize_to_dict() called
2024-11-26 13:57:13,776 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647433790899, 0)
2024-11-26 13:57:13,777 - MainThread _query_context_cache.py:180 - serialize_to_dict() - DEBUG - serialize_to_dict(): data to send to server {'entries': [{'id': 0, 'timestamp': 1732647433790899, 'priority': 0, 'context': {}}]}
2024-11-26 13:57:13,777 - MainThread connection.py:1342 - cmd_query() - DEBUG - sql=[show tables like 'GROCERY_LIST_TEST'], sequence_id=[3], is_file_transfer=[False]
2024-11-26 13:57:13,778 - MainThread network.py:488 - request() - DEBUG - Opentelemtry otel injection failed because of: No module named 'opentelemetry'
2024-11-26 13:57:13,779 - MainThread network.py:1213 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 1/1 active sessions
2024-11-26 13:57:13,779 - MainThread network.py:887 - _request_exec_wrapper() - DEBUG - remaining request timeout: N/A ms, retry cnt: 1
2024-11-26 13:57:13,780 - MainThread network.py:869 - add_request_guid() - DEBUG - Request guid: 7b902c6e-92d4-44aa-a033-e74217f2c67e
2024-11-26 13:57:13,780 - MainThread network.py:1059 - _request_exec() - DEBUG - socket timeout: 60
2024-11-26 13:57:13,902 - MainThread connectionpool.py:474 - _make_request() - DEBUG - https://account name:443 "POST /queries/v1/query-request?requestId=02a977e4-7776-4420-9a23-16cd05a46818&request_guid=7b902c6e-92d4-44aa-a033-e74217f2c67e HTTP/1.1" 200 None
2024-11-26 13:57:13,904 - MainThread network.py:1086 - _request_exec() - DEBUG - SUCCESS
2024-11-26 13:57:13,905 - MainThread network.py:1218 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 0/1 active sessions
2024-11-26 13:57:13,905 - MainThread network.py:751 - _post_request() - DEBUG - ret[code] = None, after post request
2024-11-26 13:57:13,905 - MainThread network.py:777 - _post_request() - DEBUG - Query id: 01b8a291-0412-4a5e-000c-0583ddde90f6
2024-11-26 13:57:13,906 - MainThread _query_context_cache.py:191 - deserialize_json_dict() - DEBUG - deserialize_json_dict() called: data from server: {'entries': [{'id': 0, 'timestamp': 1732647433931154, 'priority': 0}]}
2024-11-26 13:57:13,907 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647433790899, 0)
2024-11-26 13:57:13,907 - MainThread _query_context_cache.py:232 - deserialize_json_dict() - DEBUG - deserialize {'id': 0, 'timestamp': 1732647433931154, 'priority': 0}
2024-11-26 13:57:13,907 - MainThread _query_context_cache.py:101 - _sync_priority_map() - DEBUG - sync_priority_map called priority_map size = 0, new_priority_map size = 1
2024-11-26 13:57:13,907 - MainThread _query_context_cache.py:127 - trim_cache() - DEBUG - trim_cache() called. treeSet size is 1 and cache capacity is 5
2024-11-26 13:57:13,907 - MainThread _query_context_cache.py:136 - trim_cache() - DEBUG - trim_cache() returns. treeSet size is 1 and cache capacity is 5
2024-11-26 13:57:13,907 - MainThread _query_context_cache.py:271 - deserialize_json_dict() - DEBUG - deserialize_json_dict() returns
2024-11-26 13:57:13,908 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647433931154, 0)
2024-11-26 13:57:13,908 - MainThread cursor.py:995 - execute() - DEBUG - sfqid: 01b8a291-0412-4a5e-000c-0583ddde90f6
2024-11-26 13:57:13,908 - MainThread cursor.py:1001 - execute() - DEBUG - query execution done
2024-11-26 13:57:13,908 - MainThread cursor.py:1015 - execute() - DEBUG - SUCCESS
2024-11-26 13:57:13,909 - MainThread cursor.py:1034 - execute() - DEBUG - PUT OR GET: False
2024-11-26 13:57:13,909 - MainThread cursor.py:1152 - _init_result_and_meta() - DEBUG - Query result format: json
2024-11-26 13:57:13,979 - MainThread result_batch.py:487 - _parse() - DEBUG - parsing for result batch id: 1
2024-11-26 13:57:13,980 - MainThread cursor.py:1166 - _init_result_and_meta() - INFO - Number of results in first chunk: 1
2024-11-26 13:57:13,980 - MainThread server_connection.py:493 - run_query() - DEBUG - Execute query [queryID: 01b8a291-0412-4a5e-000c-0583ddde90f6] show tables like 'GROCERY_LIST_TEST'
2024-11-26 13:57:13,981 - MainThread result_set.py:87 - result_set_iterator() - DEBUG - beginning to schedule result batch downloads
2024-11-26 13:57:13,987 - MainThread cursor.py:921 - execute() - DEBUG - executing SQL/command
2024-11-26 13:57:13,987 - MainThread cursor.py:936 - execute() - DEBUG - query: [INSERT OVERWRITE INTO GROCERY_LIST_TEST("LIST_ID", "REF_LIST_ID") SELECT "LIS...]
2024-11-26 13:57:13,988 - MainThread connection.py:1654 - _next_sequence_counter() - DEBUG - sequence counter: 4
2024-11-26 13:57:13,988 - MainThread cursor.py:644 - _execute_helper() - DEBUG - Request id: e0422dc8-4da7-4d3d-bb4c-58f666143e9e
2024-11-26 13:57:13,988 - MainThread cursor.py:646 - _execute_helper() - DEBUG - running query [INSERT OVERWRITE INTO GROCERY_LIST_TEST("LIST_ID", "REF_LIST_ID") SELECT "LIS...]
2024-11-26 13:57:13,989 - MainThread cursor.py:653 - _execute_helper() - DEBUG - is_file_transfer: True
2024-11-26 13:57:13,989 - MainThread connection.py:1313 - cmd_query() - DEBUG - _cmd_query
2024-11-26 13:57:13,989 - MainThread _query_context_cache.py:155 - serialize_to_dict() - DEBUG - serialize_to_dict() called
2024-11-26 13:57:13,989 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647433931154, 0)
2024-11-26 13:57:13,990 - MainThread _query_context_cache.py:180 - serialize_to_dict() - DEBUG - serialize_to_dict(): data to send to server {'entries': [{'id': 0, 'timestamp': 1732647433931154, 'priority': 0, 'context': {}}]}
2024-11-26 13:57:13,990 - MainThread connection.py:1342 - cmd_query() - DEBUG - sql=[INSERT OVERWRITE INTO GROCERY_LIST_TEST("LIST_ID", "REF_LIST_ID") SELECT "LIS...], sequence_id=[4], is_file_transfer=[False]
2024-11-26 13:57:13,991 - MainThread network.py:488 - request() - DEBUG - Opentelemtry otel injection failed because of: No module named 'opentelemetry'
2024-11-26 13:57:13,992 - MainThread network.py:1213 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 1/1 active sessions
2024-11-26 13:57:13,992 - MainThread network.py:887 - _request_exec_wrapper() - DEBUG - remaining request timeout: N/A ms, retry cnt: 1
2024-11-26 13:57:13,992 - MainThread network.py:869 - add_request_guid() - DEBUG - Request guid: 7fc18403-93de-41f2-bd1a-e36754f57f9a
2024-11-26 13:57:13,993 - MainThread network.py:1059 - _request_exec() - DEBUG - socket timeout: 60
2024-11-26 13:57:14,465 - MainThread connectionpool.py:474 - _make_request() - DEBUG - https://account name:443 "POST /queries/v1/query-request?requestId=e0422dc8-4da7-4d3d-bb4c-58f666143e9e&request_guid=7fc18403-93de-41f2-bd1a-e36754f57f9a HTTP/1.1" 200 None
2024-11-26 13:57:14,467 - MainThread network.py:1086 - _request_exec() - DEBUG - SUCCESS
2024-11-26 13:57:14,467 - MainThread network.py:1218 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 0/1 active sessions
2024-11-26 13:57:14,468 - MainThread network.py:751 - _post_request() - DEBUG - ret[code] = 100046, after post request
2024-11-26 13:57:14,468 - MainThread network.py:777 - _post_request() - DEBUG - Query id: 01b8a291-0412-4987-000c-0583ddde8b4a
2024-11-26 13:57:14,468 - MainThread cursor.py:995 - execute() - DEBUG - sfqid: 01b8a291-0412-4987-000c-0583ddde8b4a
2024-11-26 13:57:14,469 - MainThread cursor.py:1001 - execute() - DEBUG - query execution done
2024-11-26 13:57:14,469 - MainThread cursor.py:1076 - execute() - DEBUG - {'data': {'errorCode': '100046', 'age': 0, 'sqlState': '22003', 'queryId': '01b8a291-0412-4987-000c-0583ddde8b4a'}, 'code': '100046', 'message': 'Number out of representable range: type FIXEDSB16{nullable}, value nan', 'success': False, 'headers': None}
Traceback (most recent call last):
File "C:\Users\uthusoo\tests\test_sf.py", line 40, in
snowpark_load_table_df.write.save_as_table(
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\telemetry.py", line 227, in wrap
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark\dataframe_writer.py", line 314, in save_as_table
result = session._conn.execute(
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\server_connection.py", line 598, in execute
result_set, result_meta = self.get_result_set(
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\analyzer\snowflake_plan.py", line 205, in wrap
raise ne.with_traceback(tb) from None
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\analyzer\snowflake_plan.py", line 136, in wrap
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\server_connection.py", line 707, in get_result_set
result = self.run_query(
^^^^^^^^^^^^^^^
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\server_connection.py", line 129, in wrap
raise ex
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\server_connection.py", line 123, in wrap
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\server_connection.py", line 505, in run_query
raise ex
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\server_connection.py", line 490, in run_query
results_cursor = self.execute_and_notify_query_listener(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\server_connection.py", line 434, in execute_and_notify_query_listener
raise err
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\snowpark_internal\server_connection.py", line 429, in execute_and_notify_query_listener
results_cursor = self._cursor.execute(query, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\connector\cursor.py", line 1097, in execute
Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\connector\errors.py", line 284, in errorhandler_wrapper
handed_over = Error.hand_to_other_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\connector\errors.py", line 339, in hand_to_other_handler
cursor.errorhandler(connection, cursor, error_class, error_value)
File "C:\Users\uthusoo\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\snowflake\connector\errors.py", line 215, in default_errorhandler
raise error_class(
snowflake.snowpark.exceptions.SnowparkSQLException: (1304): 01b8a291-0412-4987-000c-0583ddde8b4a: 100046 (22003): 01b8a291-0412-4987-000c-0583ddde8b4a: Number out of representable range: type FIXEDSB16{nullable}, value nan
2024-11-26 13:57:14,495 - MainThread session.py:687 - close() - INFO - Closing session: 3383760240866770
2024-11-26 13:57:14,496 - MainThread session.py:940 - cancel_all() - INFO - Canceling all running queries
2024-11-26 13:57:14,496 - MainThread cursor.py:921 - execute() - DEBUG - executing SQL/command
2024-11-26 13:57:14,496 - MainThread cursor.py:936 - execute() - DEBUG - query: [select system$cancel_all_queries(3383760240866770)]
2024-11-26 13:57:14,497 - MainThread connection.py:1654 - _next_sequence_counter() - DEBUG - sequence counter: 5
2024-11-26 13:57:14,497 - MainThread cursor.py:644 - _execute_helper() - DEBUG - Request id: 5341d619-955d-467e-bf3d-a8aba9ca275e
2024-11-26 13:57:14,497 - MainThread cursor.py:646 - _execute_helper() - DEBUG - running query [select system$cancel_all_queries(3383760240866770)]
2024-11-26 13:57:14,497 - MainThread cursor.py:653 - _execute_helper() - DEBUG - is_file_transfer: True
2024-11-26 13:57:14,498 - MainThread connection.py:1313 - cmd_query() - DEBUG - _cmd_query
2024-11-26 13:57:14,498 - MainThread _query_context_cache.py:155 - serialize_to_dict() - DEBUG - serialize_to_dict() called
2024-11-26 13:57:14,498 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647433931154, 0)
2024-11-26 13:57:14,498 - MainThread _query_context_cache.py:180 - serialize_to_dict() - DEBUG - serialize_to_dict(): data to send to server {'entries': [{'id': 0, 'timestamp': 1732647433931154, 'priority': 0, 'context': {}}]}
2024-11-26 13:57:14,498 - MainThread connection.py:1342 - cmd_query() - DEBUG - sql=[select system$cancel_all_queries(3383760240866770)], sequence_id=[5], is_file_transfer=[False]
2024-11-26 13:57:14,500 - MainThread network.py:488 - request() - DEBUG - Opentelemtry otel injection failed because of: No module named 'opentelemetry'
2024-11-26 13:57:14,500 - MainThread network.py:1213 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 1/1 active sessions
2024-11-26 13:57:14,500 - MainThread network.py:887 - _request_exec_wrapper() - DEBUG - remaining request timeout: N/A ms, retry cnt: 1
2024-11-26 13:57:14,501 - MainThread network.py:869 - add_request_guid() - DEBUG - Request guid: b78d6947-fda0-4289-bbe6-fb2d7dbc5e33
2024-11-26 13:57:14,501 - MainThread network.py:1059 - _request_exec() - DEBUG - socket timeout: 60
2024-11-26 13:57:14,572 - MainThread connectionpool.py:474 - _make_request() - DEBUG - https://account name:443 "POST /queries/v1/query-request?requestId=5341d619-955d-467e-bf3d-a8aba9ca275e&request_guid=b78d6947-fda0-4289-bbe6-fb2d7dbc5e33 HTTP/1.1" 200 None
2024-11-26 13:57:14,574 - MainThread network.py:1086 - _request_exec() - DEBUG - SUCCESS
2024-11-26 13:57:14,574 - MainThread network.py:1218 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 0/1 active sessions
2024-11-26 13:57:14,575 - MainThread network.py:751 - _post_request() - DEBUG - ret[code] = None, after post request
2024-11-26 13:57:14,575 - MainThread network.py:777 - _post_request() - DEBUG - Query id: 01b8a291-0412-3d32-000c-0583ddde6e66
2024-11-26 13:57:14,575 - MainThread _query_context_cache.py:191 - deserialize_json_dict() - DEBUG - deserialize_json_dict() called: data from server: {'entries': [{'id': 0, 'timestamp': 1732647434583269, 'priority': 0}]}
2024-11-26 13:57:14,576 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647433931154, 0)
2024-11-26 13:57:14,576 - MainThread _query_context_cache.py:232 - deserialize_json_dict() - DEBUG - deserialize {'id': 0, 'timestamp': 1732647434583269, 'priority': 0}
2024-11-26 13:57:14,577 - MainThread _query_context_cache.py:101 - _sync_priority_map() - DEBUG - sync_priority_map called priority_map size = 0, new_priority_map size = 1
2024-11-26 13:57:14,577 - MainThread _query_context_cache.py:127 - trim_cache() - DEBUG - trim_cache() called. treeSet size is 1 and cache capacity is 5
2024-11-26 13:57:14,577 - MainThread _query_context_cache.py:136 - trim_cache() - DEBUG - trim_cache() returns. treeSet size is 1 and cache capacity is 5
2024-11-26 13:57:14,578 - MainThread _query_context_cache.py:271 - deserialize_json_dict() - DEBUG - deserialize_json_dict() returns
2024-11-26 13:57:14,578 - MainThread _query_context_cache.py:276 - log_cache_entries() - DEBUG - Cache Entry: (0, 1732647434583269, 0)
2024-11-26 13:57:14,579 - MainThread cursor.py:995 - execute() - DEBUG - sfqid: 01b8a291-0412-3d32-000c-0583ddde6e66
2024-11-26 13:57:14,579 - MainThread cursor.py:1001 - execute() - DEBUG - query execution done
2024-11-26 13:57:14,579 - MainThread cursor.py:1015 - execute() - DEBUG - SUCCESS
2024-11-26 13:57:14,580 - MainThread cursor.py:1034 - execute() - DEBUG - PUT OR GET: False
2024-11-26 13:57:14,580 - MainThread cursor.py:1152 - _init_result_and_meta() - DEBUG - Query result format: arrow
2024-11-26 13:57:14,580 - MainThread cursor.py:1166 - _init_result_and_meta() - INFO - Number of results in first chunk: 1
2024-11-26 13:57:14,581 - MainThread server_connection.py:493 - run_query() - DEBUG - Execute query [queryID: 01b8a291-0412-3d32-000c-0583ddde6e66] select system$cancel_all_queries(3383760240866770)
2024-11-26 13:57:14,581 - MainThread result_batch.py:68 - _create_nanoarrow_iterator() - DEBUG - Using nanoarrow as the arrow data converter
2024-11-26 13:57:14,581 - MainThread CArrowIterator.cpp:120 - CArrowIterator() - DEBUG - Arrow BatchSize: 1
2024-11-26 13:57:14,582 - MainThread CArrowChunkIterator.cpp:46 - CArrowChunkIterator() - DEBUG - Arrow chunk info: batchCount 1, columnCount 1, use_numpy: 0
2024-11-26 13:57:14,582 - MainThread nanoarrow_arrow_iterator.cp311-win_amd64.pyd:0 - cinit() - DEBUG - Batches read: 0
2024-11-26 13:57:14,582 - MainThread result_set.py:87 - result_set_iterator() - DEBUG - beginning to schedule result batch downloads
2024-11-26 13:57:14,583 - MainThread CArrowChunkIterator.cpp:70 - next() - DEBUG - Current batch index: 0, rows in current batch: 1
2024-11-26 13:57:14,584 - MainThread connection.py:789 - close() - INFO - closed
2024-11-26 13:57:14,584 - MainThread telemetry.py:211 - close() - DEBUG - Closing telemetry client.
2024-11-26 13:57:14,586 - MainThread telemetry.py:176 - send_batch() - DEBUG - Sending 1 logs to telemetry. Data is {'logs': [{'message': {'driver_type': 'PythonConnector', 'driver_version': '3.12.3', 'source': 'PythonSnowpark', 'type': 'client_imported_packages', 'value': "{'pythoncom', 'linecache', 'sys', 'idna', 'reprlib', 'io', 'pathlib', 'packaging', 're', 'posixpath', 'tokenize', 'contextvars', 'pkgutil', 'pickle', 'bz2', 'requests', 'glob', 'token', 'hashlib', 'string', 'uuid', 'ssl', 'difflib', 'unittest', 'copy', 'ctypes', 'quopri', 'socket', 'abc', 'sortedcontainers', 'cmath', 'platformdirs', 'urllib', 'struct', 'win32api', 'typing', 'cryptography', 'locale', 'cython_runtime', 'tomlkit', 'builtins', 'dis', 'OpenSSL', 'dataclasses', 'weakref', 'functools', 'math', 'os', 'unicodedata', 'mmap', 'select', 'pywin32_bootstrap', 'nt', 'secrets', 'pytz', 'fnmatch', 'zipfile', 'tarfile', 'email', 'yaml', 'platform', 'pkg_resources', 'asyncio', 'concurrent', 'copyreg', 'logging', 'binascii', 'signal', 'filelock', 'opcode', 'stringprep', 'snowflake', 'certifi', 'pprint', 'textwrap', 'pywin32_system32', 'html', 'mimetypes', 'asn1crypto', 'numbers', 'statistics', 'six', 'warnings', 'time', 'stat', 'atexit', 'subprocess', 'msvcrt', 'json', 'bisect', 'gettext', 'operator', 'datetime', 'win32com', 'tzlocal', 'marshal', 'ntpath', 'queue', 'gc', 'itertools', 'zlib', 'http', 'nturl2path', 'jwt', 'site', 'contextlib', 'xml', 'collections', 'hmac', 'ipaddress', 'plistlib', 'fractions', 'genericpath', 'codecs', 'dateutil', 'heapq', 'urllib3', 'pyexpat', 'inspect', 'gzip', 'typing_extensions', 'calendar', 'webbrowser', 'numpy', 'ast', 'base64', 'shutil', 'decimal', 'argparse', 'csv', 'keyword', 'zipimport', 'array', 'threading', 'pywintypes', 'sysconfig', 'traceback', 'types', 'cloudpickle', 'zoneinfo', 'winreg', 'selectors', 'pandas', 'tempfile', 'charset_normalizer', 'random', 'shlex', 'importlib', 'errno', 'lzma', 'encodings', 'enum'}"}, 'timestamp': '1732647433647'}, {'message': {'source': 'PythonSnowpark', 'version': '1.25.0', 'python_version': '3.11.9', 'operating_system': 'Windows', 'type': 'snowpark_session_created', 'data': {'start_time': 1732647433647, 'created_by_snowpark': 1}}, 'timestamp': '1732647433647'}, {'message': {'source': 'PythonSnowpark', 'version': '1.25.0', 'python_version': '3.11.9', 'operating_system': 'Windows', 'type': 'snowpark_cursor_created', 'data': {'session_id': 3383760240866770, 'thread_ident': 10668}}, 'timestamp': '1732647433647'}, {'message': {'driver_type': 'PythonConnector', 'driver_version': '3.12.3', 'source': 'PythonSnowpark', 'type': 'client_time_consume_first_result', 'query_id': '01b8a291-0412-3d32-000c-0583ddde6e62', 'value': -18}, 'timestamp': '1732647433713'}, {'message': {'driver_type': 'PythonConnector', 'driver_version': '3.12.3', 'source': 'PythonSnowpark', 'type': 'client_time_consume_first_result', 'query_id': '01b8a291-0412-4919-000c-0583ddde7d9e', 'value': -27}, 'timestamp': '1732647433773'}, {'message': {'driver_type': 'PythonConnector', 'driver_version': '3.12.3', 'source': 'PythonSnowpark', 'type': 'client_time_consume_first_result', 'query_id': '01b8a291-0412-4a5e-000c-0583ddde90f6', 'value': -29}, 'timestamp': '1732647433908'}, {'message': {'driver_type': 'PythonConnector', 'driver_version': '3.12.3', 'source': 'PythonSnowpark', 'type': 'client_time_consume_last_result', 'query_id': '01b8a291-0412-4a5e-000c-0583ddde90f6', 'value': 73}, 'timestamp': '1732647433981'}, {'message': {'driver_type': 'PythonConnector', 'driver_version': '3.12.3', 'source': 'PythonSnowpark', 'Stacktrace': (False, ' File "snowflake\connector\cursor.py", line 1097, in execute\n File "snowflake\connector\errors.py", line 284, in errorhandler_wrapper\n File "snowflake\connector\errors.py", line 339, in hand_to_other_handler\n File "snowflake\connector\errors.py", line 215, in default_errorhandler\n', None), 'query_id': '01b8a291-0412-4987-000c-0583ddde8b4a', 'sql_state': '22003', 'reason': '100046 (22003)', 'ErrorNumber': '100046', 'type': 'client_sql_exception', 'exception': 'ProgrammingError'}, 'timestamp': '1732647434473'}, {'message': {'driver_type': 'PythonConnector', 'driver_version': '3.12.3', 'source': 'PythonSnowpark', 'type': 'client_time_consume_first_result', 'query_id': '01b8a291-0412-3d32-000c-0583ddde6e66', 'value': -28}, 'timestamp': '1732647434579'}, {'message': {'driver_type': 'PythonConnector', 'driver_version': '3.12.3', 'source': 'PythonSnowpark', 'type': 'client_time_consume_last_result', 'query_id': '01b8a291-0412-3d32-000c-0583ddde6e66', 'value': 4}, 'timestamp': '1732647434583'}, {'message': {'source': 'PythonSnowpark', 'version': '1.25.0', 'python_version': '3.11.9', 'operating_system': 'Windows', 'type': 'snowpark_temp_table_cleanup', 'data': {'session_id': 3383760240866770, 'temp_table_cleaner_enabled': False, 'num_temp_tables_cleaned': 0, 'num_temp_tables_created': 0}}, 'timestamp': '1732647434583'}]}.
2024-11-26 13:57:14,590 - MainThread network.py:488 - request() - DEBUG - Opentelemtry otel injection failed because of: No module named 'opentelemetry'
2024-11-26 13:57:14,590 - MainThread network.py:1213 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 1/1 active sessions
2024-11-26 13:57:14,590 - MainThread network.py:887 - _request_exec_wrapper() - DEBUG - remaining request timeout: 5000 ms, retry cnt: 1
2024-11-26 13:57:14,591 - MainThread network.py:869 - add_request_guid() - DEBUG - Request guid: 4866385c-4242-478f-9d18-c4f36c0cf5b0
2024-11-26 13:57:14,591 - MainThread network.py:1059 - _request_exec() - DEBUG - socket timeout: 60
2024-11-26 13:57:14,618 - MainThread connectionpool.py:474 - _make_request() - DEBUG - https://account name:443 "POST /telemetry/send?request_guid=4866385c-4242-478f-9d18-c4f36c0cf5b0 HTTP/1.1" 200 None
2024-11-26 13:57:14,620 - MainThread network.py:1086 - _request_exec() - DEBUG - SUCCESS
2024-11-26 13:57:14,620 - MainThread network.py:1218 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 0/1 active sessions
2024-11-26 13:57:14,620 - MainThread network.py:751 - _post_request() - DEBUG - ret[code] = None, after post request
2024-11-26 13:57:14,621 - MainThread telemetry.py:200 - send_batch() - DEBUG - Successfully uploading metrics to telemetry.
2024-11-26 13:57:14,621 - MainThread connection.py:795 - close() - INFO - No async queries seem to be running, deleting session
2024-11-26 13:57:14,621 - MainThread network.py:1213 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 1/1 active sessions
2024-11-26 13:57:14,622 - MainThread network.py:887 - _request_exec_wrapper() - DEBUG - remaining request timeout: 5000 ms, retry cnt: 1
2024-11-26 13:57:14,622 - MainThread network.py:869 - add_request_guid() - DEBUG - Request guid: a42cb063-24a6-43b2-a0b0-ac4c4039e47c
2024-11-26 13:57:14,623 - MainThread network.py:1059 - _request_exec() - DEBUG - socket timeout: 60
2024-11-26 13:57:14,664 - MainThread connectionpool.py:474 - _make_request() - DEBUG - https://account name:443 "POST /session?delete=true&request_guid=a42cb063-24a6-43b2-a0b0-ac4c4039e47c HTTP/1.1" 200 None
2024-11-26 13:57:14,665 - MainThread network.py:1086 - _request_exec() - DEBUG - SUCCESS
2024-11-26 13:57:14,666 - MainThread network.py:1218 - _use_requests_session() - DEBUG - Session status for SessionPool 'account name', SessionPool 0/1 active sessions
2024-11-26 13:57:14,666 - MainThread network.py:751 - _post_request() - DEBUG - ret[code] = None, after post request
2024-11-26 13:57:14,668 - MainThread _query_context_cache.py:141 - clear_cache() - DEBUG - clear_cache() called
2024-11-26 13:57:14,668 - MainThread connection.py:808 - close() - DEBUG - Session is closed
2024-11-26 13:57:14,669 - MainThread session.py:695 - close() - INFO - Closed session: 3383760240866770

import logging

for logger_name in ('snowflake.snowpark', 'snowflake.connector'):
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
    logger.addHandler(ch)
@uthusoo uthusoo added bug Something isn't working needs triage Initial RCA is required labels Nov 26, 2024
@github-actions github-actions bot changed the title Snowpark not able to handle python None insertion to snowflake tables SNOW-1830018: Snowpark not able to handle python None insertion to snowflake tables Nov 26, 2024
@sfc-gh-sghosh sfc-gh-sghosh self-assigned this Nov 27, 2024
@sfc-gh-sghosh sfc-gh-sghosh added status-triage Issue is under initial triage and removed needs triage Initial RCA is required labels Nov 27, 2024
@sfc-gh-sghosh
Copy link

Hello @uthusoo ,

Snowflake's NUMBER column does not allow NaN values. Instead, it expects NULL for missing numeric values.
When you use session.create_dataframe(), Snowpark translates the NaN from Pandas into a numeric value that Snowflake cannot store (NaN is outside the valid range for NUMBER).

So, as a workaround, please explicitly replace None/NaN in the DataFrame with NULL (or an equivalent value) before writing to Snowflake.

Example: Try below code snippet

`

session.sql(
"create or replace TABLE GROCERY_LIST_TEST (LIST_ID NUMBER(26,0) NOT NULL ,REF_LIST_ID NUMBER(26,0))"
).collect()

data = pd.DataFrame([[1, None], [2, 2], [3, 3]], columns=["LIST_ID", "REF_LIST_ID"])

--Replace None with NoneType equivalent in Pandas
data["REF_LIST_ID"] = data["REF_LIST_ID"].astype("Int64") # Nullable integer type

snowpark_load_table_df = session.create_dataframe(data)

table_name = "GROCERY_LIST_TEST"
mode = "truncate"
column_order = "name"

snowpark_load_table_df.write.save_as_table(table_name, mode=mode, column_order=column_order)

t = session.table("GROCERY_LIST_TEST")
t.show()

output:

|"LIST_ID" |"REF_LIST_ID" |

|1 |NULL |
|2 |2 |
|3 |3 |

`

@sfc-gh-sghosh sfc-gh-sghosh added status-triage_done Initial triage done, will be further handled by the driver team and removed bug Something isn't working status-triage Issue is under initial triage labels Nov 27, 2024
@uthusoo
Copy link
Author

uthusoo commented Nov 27, 2024

It Worked, Thanks a lot @sfc-gh-sghosh. This had been a big issue, and I could not find any solution on the internet with the dataframe.

@uthusoo uthusoo closed this as completed Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sfc-gh-sghosh sfc-gh-sghosh

Labels
status-triage_done Initial triage done, will be further handled by the driver team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sfc-gh-sghosh @uthusoo