How to escape "=" character in url ?

[SimonYang-CS]

Hello team,

I want to set a Var key=value in Kyuubi JDBC Url, but the value is a base64 string, that contains "=" character .

How to escape "=" character in url var ?

url: "fs.azure.account.key.pfsdpstorage.dfs.core.windows.net=............A=="

run "set " command:

+---------------------------------------------------------------------------+----------------------------------------------------+
| key | value |
+---------------------------------------------------------------------------+----------------------------------------------------+
| fs.azure.account.key.pfsdpstorage.dfs.core.windows.net=............A= | |

Expected:

+---------------------------------------------------------------------------+----------------------------------------------------+
| key | value |
+---------------------------------------------------------------------------+----------------------------------------------------+
| fs.azure.account.key.pfsdpstorage.dfs.core.windows.net= | ............A== |

i try to replace "=" with %3D, not luck.

The only work around:

spark.conf.set("fs.azure.account.key.pfsdpstorage.dfs.core.windows.net", "  ............A==");
spark.sql("SET kyuubi.operation.language=SQL;");

[yaooqinn]

seems it needs to be handled on our jdbc driver side. cc @turboFei.

Can you try to remove the trailing '=' padding? @SimonYang-CS

[yaooqinn]

It depends on which JDBC driver you are using, if you use the one we provided, you can check KyuubiConnection

[SimonYang-CS]

I use the beeline from spark distribution( spark/bin/beeline), I think it is default hive JDBC driver.

[yaooqinn]

then you need to fix it in hive-jdbc module in the Apache Hive repo, or using KyuubiConnection instead and fix it in kyuubi-hive-jdbc module

[cxzl25]

beeline -u "jdbc:hive2://xxxx" --hivevar a=abc=

set a;

+------+--------+
| key  | value  |
+------+--------+
| a    | abc=   |
+------+--------+

[SimonYang-CS]

thanks all for help. for a quick around, I choose cxz125's suggestion.

I looked into the source code:

https://github.com/apache/incubator-kyuubi/blob/master/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/Utils.java#L361

Due to the regular express can't handle the case.

[wForget]

You can try url encoding, = to %3D

[SimonYang-CS]

I try %3D, no luck.

[wForget]

seems to cover all possible vars

jdbc:hive2://XXXX?spark.key.test1=test1=test1#spark.key.test2=test2=test2

run set:

[SimonYang-CS]

@wForget , in our case, the key name is fs.azure.account.key.pfsdpstorage.dfs.core.windows.net. it doesn't have the prefix: spark.

[SimonYang-CS]

I looked into beeline source, I think we can pass the key=var from Properties.

  public Connection getConnectionFromLocalDriver(String url, Properties properties) {
    Collection<Driver> drivers = beeLine.getDrivers();
    for (Driver d : drivers) {
      try {
        if (d.acceptsURL(url) && beeLine.isSupportedLocalDriver(d)) {
          String clazzName = d.getClass().getName();
          beeLine.debug("Driver name is " + clazzName);
          Driver driver =
            (Driver) Class.forName(clazzName, true, Thread.currentThread().getContextClassLoader())
              .newInstance();
          return driver.connect(url, properties);
        }
      } catch (Exception e) {
        beeLine.error("Fail to connect with a local driver due to the exception:" + e);
        beeLine.error(e);
      }
    }
    return null;
  }