Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can we connect to proxy server using com.springml.spark.salesforce driver #80

Open
Brooody opened this issue Apr 24, 2023 · 0 comments
Open

Comments

@Brooody
Copy link

Brooody commented Apr 24, 2023

I am trying to connect to Salesforce url(login.salesforce.com) using a proxy server in pyspark with com.springml.spark.salesforce driver.

I tried passing/adding the proxy server details in the config file, in the spark submit command and even while using the interactive mode but nothing worked. Wanted to check if the proxies are accepted by the com.springml.spark.salesforce driver. I am using this package: spark-salesforce_2.12-1.1.4.jar

Spark version: 3.0.0 Scala Version: 2.12.10 python: 3.6.8

While Execution:

Spark.sparkContext.setSystemProperty("http.proxyHost", "proxy")
spark.sparkContext.setSystemProperty("http.proxyPort", "proxyPort")
spark.sparkContext.setSystemProperty("https.proxyHost", "proxy")
spark.sparkContext.setSystemProperty("https.proxyPort", "proxyPort")
spark.sparkContext.setSystemProperty("http.useProxy", "true")

Config file:

.set("spark.driver.extraJavaOptions", "-Dhttp.proxyHost=proxy -Dhttp.proxyPort=proxyPort -Dhttps.proxyHost=proxy -Dhttps.proxyPort=proxyPort") \ 
.set("spark.executor.extraJavaOptions", "-Dhttp.proxyHost=proxy -Dhttp.proxyPort=proxyPort -Dhttps.proxyHost=proxy -Dhttps.proxyPort=proxyPort"")

code:

import sys
from pyspark.sql import SparkSession
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql.functions import concat, col, lit

spark.sparkContext.setSystemProperty("http.proxyHost", "proxyHost")
spark.sparkContext.setSystemProperty("http.proxyPort", "proxyPort")
spark.sparkContext.setSystemProperty("https.proxyHost", "proxyHost")
spark.sparkContext.setSystemProperty("https.proxyPort", "proxyPort")

df = spark \
     .read \
     .format("com.springml.spark.salesforce")\
     .option("dbtable","dbtable")\
     .option("username", "username") \
     .option("password","password" )\
     .option("login","https://login.salesforce.com") \
     .option("version","44.0")\
     .option("sfObject","sfObject")\
     .option("soql", query) \
     .load()

Error:

py4j.protocol.Py4JJavaError: An error occurred while calling o119.load.
: java.lang.Exception: com.sforce.ws.ConnectionException: Failed to send request to https://login.salesforce.com/services/Soap/u/35.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant