SNOW-1062400: Failing to upload a pickled item from native app to a (non-temporary) stage

[arghhjayy]

Please answer these questions before submitting your issue. Thanks!

1. What version of Python are you using? - 3.8

2. What operating system and processor architecture are you using? - Streamlit Apps in Snowsight

3. What are the component versions in the environment (pip freeze)? - Snowflake python connector and Streamlit

4. What did you do? -
   The original problem was to save a pickled item to a (non-temporary) stage inside a native-app.
   Say I have a Streamlit native app with two pages.
   Page 1 -> trains the model and stores it somewhere
   Page 2 -> uses the stored model from page1 to perform some action

Page 1 pseudocode:

# Note: the object graph is not a string here, object graph_string is
graph_dict = {
    "graph": graph,
    "graph_string": graph_string,
    .
    .
}

input_stream = io.BytesIO()
dump(graph_dict, input_stream)

put_result = session.file.put_stream(
    input_stream, f"@STAGE_NAME/{model_name}.joblib", overwrite=True
)

Page 2:

from joblib import load
model_path = f"@STAGE_NAME/" + model_name + ".joblib.gz"
output_stream = session.file.get_stream(model_path)
model_dict = load(output_stream)

When we do this in our local development env, it works well but when we promote the same nativeapp, it throws an error when we try to upload it to the stage.

Here's the stack trace (from nativeapp):

SystemError: <built-in function open_stream> returned a result with an error set
Traceback:
File "/usr/lib/python_udf/23585920156cce9b4493606805779aacfceaa197754fbbe35e87421ddbf97d05/lib/python3.8/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)
File "/home/udf/10738074217/streamlit_app.py", line 53, in <module>
    put_result = session.file.put_stream(
File "/usr/lib/python_udf/23585920156cce9b4493606805779aacfceaa197754fbbe35e87421ddbf97d05/lib/python3.8/site-packages/snowflake/snowpark/file_operation.py", line 267, in put_stream
    cursor._upload_stream(input_stream, stage_location, options)
File "/usr/lib/python_udf/23585920156cce9b4493606805779aacfceaa197754fbbe35e87421ddbf97d05/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 606, in _upload_stream
    file_transfer_agent.execute()
File "/usr/lib/python_udf/23585920156cce9b4493606805779aacfceaa197754fbbe35e87421ddbf97d05/lib/python3.8/site-packages/snowflake/connector/file_transfer_agent.py", line 166, in execute
    self.transfer(self._file_metadata)
File "/usr/lib/python_udf/23585920156cce9b4493606805779aacfceaa197754fbbe35e87421ddbf97d05/lib/python3.8/site-packages/snowflake/connector/file_transfer_agent.py", line 181, in transfer
    file.upload()
File "/usr/lib/python_udf/23585920156cce9b4493606805779aacfceaa197754fbbe35e87421ddbf97d05/lib/python3.8/site-packages/snowflake/connector/stored_proc_storage_client.py", line 122, in upload
    wstream = _sfstream.SfStream(
File "_sfstream.py", line 49, in __init__

6. What did you expect to see? - It should upload to the stage in nativeapp, like it did in local development

7. Can you set logging to DEBUG and collect the logs?

   import logging
   
   for logger_name in ('snowflake.snowpark', 'snowflake.connector'):
       logger = logging.getLogger(logger_name)
       logger.setLevel(logging.DEBUG)
       ch = logging.StreamHandler()
       ch.setLevel(logging.DEBUG)
       ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
       logger.addHandler(ch)
   

Also, I see no nativeapp tag on here