Support HTTP Proxies

[tombentley]

The URL/Dav content stores should support accessing the net via a proxy. We should also support proxy authentication.

[FroMage]

That needs support from the config file too, right?

[tombentley]

It will do, yeah. I'd imagine something like this:

[proxy]
host=myproxy.mynetwork
port=1234
non-proxy-hosts=foo.mynetwork, bar.mynetwork
# EITHER: plaintext password
password=myPlaintextPassword
# OR: password in a keystore
keystore-alias=proxy-password

[tombentley]

Wow, you have to love java.net. This page has some useful info: http://docs.oracle.com/javase/6/docs/technotes/guides/net/proxies.html Seems you can use the system's proxy settings (on Windows and Gnome anyway), though this only works at the ProxySelector level, and only if you set the right system property. I think we should use the system's proxy settings by default, which would mean:

• Setting the system property, and
• Setting the system-wide ProxySelector.setDefault() if a [proxy] was specified.

Then there's the question of which proxy settings win out in the IDE (the ones in the config file, or on the IDE).

[tombentley]

ceylon-common now supports the configuration of a proxy. For the built injava.net protocol clients it should be a matter of calling Authentication.installProxy() to setup a java.net.ProxySelector (and a java.net.Authenticator if the proxy requires auth).

[FroMage]

So does this mean this is resolved?

[tombentley]

Alas, not yet. We still need to configure CMR to use the proxy.

[FroMage]

Well, moving to M6.

[FroMage]

Moving again.

[quintesse]

Ok, this wasn't as straight-forward as I hoped it was due to Java's brain dead implementation of this. You can only set authentication globally for the JVM? Reallly?

So a real solution might require the use of Apache's HttpComponents (which we're already including for Maven support, but maybe we'd like that to be as optional as possible) or write our own authentication handling (no idea how hard that would be, "Basic" proxy authentication is easy enough, but what else is there?)

Anyway, what I pushed should make non-authenticating proxies work. No tests though.

[tombentley]

"Basic" proxy authentication is easy enough, but what else is there?

I've encountered a proxy which used NTLM authentication, which is a barrel of laughs to deal with in Java.

[quintesse]

I've encountered a proxy which used NTLM authentication

And does Java's Authentication even deal with that?

[tombentley]

It was a long time ago, thankfully, so I've largely forgotten how I got around it, but I think the gist of the answer is "no, not in any way". NTLM was old hat even then, so I don't seriously propose we worry about it, but I would guess that something like Kerberos would still be current in corporate environments.

[quintesse]

So a temporary hack/work-around would be to install the proxy handler if somebody passed -Dceylon.installGlobalProxyHandler=true on the command line. It's crappy but it would at least make things work for now and not mess with embedded use-cases. WDYT @FroMage @tombentley ?

[ThiporKong]

Refactor access to URL.openConnection into some interface with two strategies, one JDK-based and one HttpClient-based. Make the JDK-based one the default to avoid the dependency on HttpClient.
Turn the the HttpClient-based strategy on, if proxy.{user,password} properties are set or have a proxy.connectionStrategyClass=... setting.

[josephwinston]

I'm using ceylon version 1.2.0 (A Series Of Unlikely Explanations) on Kubuntu 14.04 behind a ntlm proxy. I have a cntlm version running in linux.

Finding the proxy discusssion, I added to ~/.ceylon/config:

# define a proxy for accessing the network
[proxy]
host=localhost
port=3128

But I cannot access http://repo1.maven.org/maven2/. A typical error message is:

Warning: Failed downloading org/apache/spark/spark-core_2.10/1.2.1/spark-core_2.10-1.2.1.pom from http://repo1.maven.org/maven2/. Reason: 
org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.apache.spark:spark-core_2.10:pom:1.2.1 from/to central (http://repo1.maven.org/maven2): Error transferring file: Connection refused from http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10/1.2.1/spark-core_2.10-1.2.1.pom 
/disk2/hb55683/work/DistributedArchitecture/CEYOLN/interop-java/source/com/example/interop/module.ceylon:22: error: cannot find module artifact org.apache.spark:spark-core_2.10-1.2.1(.car|.jar)

However, wget works with cntlm:

get http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10/1.2.1/spark-core_2.10-1.2.1.pom
--2015-11-06 13:38:26--  http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10/1.2.1/spark-core_2.10-1.2.1.pom
Connecting to 127.0.0.1:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 18595 (18K) [text/xml]
Saving to: ‘spark-core_2.10-1.2.1.pom’

100%[====================================================================================================================>] 18,595      --.-K/s   in 0.02s   

2015-11-06 13:38:26 (1.03 MB/s) - ‘spark-core_2.10-1.2.1.pom’ saved [18595/18595]

Suggestions welcome.

[quintesse]

@josephwinston Maven has its own proxies that I think you'll need to configure, see here for more info: https://maven.apache.org/guides/mini/guide-proxies.html

[josephwinston]

My ~/.m2/settings.xml are in place and work for maven.

[quintesse]

Could you try and see what happens when you specifically point Ceylon to that file by adding --aether:/path/to/settings.xml to the command line while compiling?