Operability behind a proxy

[dapdizzy]

Right now, elixir-slack based bots do not work behind a proxy.
The networking modules that Elixir-Slack is based on, are not passing proxy and proxy_auth options that are required to enable the app behind the prxy. This significantly limits the potential usage of Elixir-Slack bots.

[BlakeWilliams]

Can you be more specific? Is it the websocket connection? Initial authentication with Slack?

[dapdizzy]

I probably can, but that requires digging in.
At first it's https connection (initial connection), then, probably, ssl websocket interaction.
The thing is that 'proxy' needs to be bypassed throughout the whole network related stack which is not the case now. I could probably dig deeper, but that would require some tracing and related digging stuff.

For now, it looks to me like this:

1. 1. Initial connection requires bypassing proxy parameters (options)
2. Consecutive websocket interaction probably needs 'proxy' knwowledge too.

[dapdizzy]

When you run the elixir-slack based app behind the proxy (current code) it exits with a timeout. When I pass 'proxy' options to the initial connection call, it gies a little bit further and exits with some other error.
I believe this 'user defined issue' could be the basis for working on implementing operability behind a proxy. As I mentioned I would be able to provide specific details on the issue as the process goes..
The problem I faced when trying to implement proxied behavior was somewhere inside websocket_client erlang library, when I tried to pass proxy options to be applied in websocket_client for ssl erlang module calls.

[BlakeWilliams]

Interesting.

This isn't something I'm running into, so I'm going to close this for now. I'd happily take a PR that makes this work without changing too much code. If you decide to tackle it, feel free to reopen or open a new issue.

[splashx]

This issue is still existing while trying to run the example in a machine (Ubuntu 16.04) which can only go out to the internet via proxy:

iex(1)> Slack.Bot.start_link(SlackRtm, [], "xoxb-xxxxxxxxx")
{:error, "Timed out while connecting to the Slack RTM API"}

$ export | grep -i PROXY
declare -x http_proxy="http://100.120.70.70:3128"
declare -x https_proxy="http://100.120.70.70:3128"

$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
http_proxy=http://100.120.70.70:3128
https_proxy=http://100.120.70.70:3128

@BlakeWilliams Could you please reopen the issue?

[dapdizzy]

Of course it does. Author just ignored the description and pretended it was all good. The whole tcp/ssl stack does not make use of the proxy parameters (just as I initially described it). Сб, 29 июля 2017 г. в 15:11, Diogenes S. Jesus <[email protected]>:

…

This issue is still existing when trying to run the example in a machine (Ubuntu 16.04) which can only go out to the internet via proxy: iex(1)> Slack.Bot.start_link(SlackRtm, [], "xoxb-xxxxxxxxx") {:error, "Timed out while connecting to the Slack RTM API"} $ export | grep -i PROXY declare -x http_proxy="http://100.120.70.70:3128" declare -x https_proxy="http://100.120.70.70:3128" $ cat /etc/environment PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games" http_proxy=http://100.120.70.70:3128 https_proxy=http://100.120.70.70:3128 Could you please reopen the issue? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD4VeCa4RuLoZan5IyMHPVm3FRkAET9Rks5sSyGHgaJpZM4LPGd8> .

[splashx]

httpoison supports it, so it's possible to be implemented OoTB in these .ex files.

I don't even think it's required to check "when proxy is in env" - an empty value for :proxy will totally ignore it. The standard for pretty much any sofware and language is to look up http_proxy/HTTP_PROXY and/or https_proxy/HTTPS_PROXY from System env..

[dapdizzy]

Httpoison is not enough. You should handle socket communication as well. AFAIR, it turned out to be not implemented for the underlying Erlang library (proxy parameters were not passed where needed). Сб, 29 июля 2017 г. в 15:22, Diogenes S. Jesus <[email protected]>:

…

httpoison supports it <https://github.com/edgurgel/httpoison/blob/bd6d601c38aa5570d633f4eada6a94c1e051a8b7/test/httpoison_base_test.exs#L96-L104>, so it's possible to be implemented OoTB. I don't even think it's required to check "when proxy is in env" - an empty value for :proxy will totally ignore it. The standard for pretty much any sofware and language is http_proxy/HTTP_PROXY and https_proxy/ HTTPS_PROXY. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD4VeFih3e38Ax5DmzaUtamlMiE0vnERks5sSyQegaJpZM4LPGd8> .

[splashx]

I've gone as far as hitting this issue in the websockets_client project. Apparently abandoned :(

[dapdizzy]

Right, but it doesn't seem hard to bypass proxy options into websocket_client, I messed up building it with Rebar and stopped there. May be we could have a try and fix the issue in websocket_client? Сб, 29 июля 2017 г. в 16:43, Diogenes S. Jesus <[email protected]>:

…

I've gone as far as hitting this issue <jeremyong/websocket_client#46> in the websockets_client project. Apparently abandoned :( — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD4VeBFh5sN0H8DVl3Gv-b9XZlKEcZbVks5sSzccgaJpZM4LPGd8> .

[splashx]

@dapdizzy if you take a look at the project fork timeline one wonders if we should AT ALL deal with @jeremyong - maybe it's worth using @sanmiguel (meaning it still lives) - I've opened an issue there too. @BlakeWilliams what's your opinion on that?

[splashx]

Update: apparently it's already upstream via sanmiguel's so let's move the discussion there.

[dapdizzy]

Great! Loop me in the discussion if I'm not on the list. It'll be pretty satisfying to finally resolve the issue for good. Сб, 29 июля 2017 г. в 17:14, Diogenes S. Jesus <[email protected]>:

…

Update: apparently <https://hex.pm/packages/websocket_client> it's already upstream via sanmiguel's so let's move the discussion there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD4VeOA7jaf3eFNfuyuSyAW9qb_Nk__wks5sSz4xgaJpZM4LPGd8> .

[BlakeWilliams]

Of course it does. Author just ignored the description and pretended it was
all good.

This isn't a problem I have personally ran into so I don't have much interest in tracking it down honestly. Like I mentioned before, I'm happy to accept a PR for it and maintain it.

We're using @sanmiguel's fork since it's actively maintained and has a lot of improvements over the original. Tracking the issue you opened and seeing where that goes is likely the first step in getting this resolved.

[splashx]

@BlakeWilliams I understand you have not faced the issue, but it exists - having it marked as closed is most likely not the best feedback for those relying on the lib and running apps behind corporate proxies, though, because the issue exists.

[dapdizzy]

Right, I forgot about that kind of 'go ahead with a PR' proposal, so closing that way wasn't really that bad. At least now we have an accurate view of the problem and hopefully we'll drive it to a positive resolution. Сб, 29 июля 2017 г. в 17:41, Diogenes S. Jesus <[email protected]>:

…

@BlakeWilliams <https://github.com/blakewilliams> I understand you have not faced the issue, but it exists - having it marked as closed is most likely not the best feedback for those relying on the lib and running apps behind corporate proxies, though, because the issue exists. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD4VeKDvSSgRwgrTNQA6JBYJq2XxE9Sgks5sS0STgaJpZM4LPGd8> .

[BlakeWilliams]

@splashx That's fair, it also seemed like the issue was somewhat dead until today. :)

I just reopened it since there seems to be progress.

[dapdizzy]

I'm debugging the issue right now and it seems to fail on the call to ssl:connect/4 in websocket_client library.
The Proxy itself it pibked up from the http_proxy environment variable (which can be set in the command line prompt before starting the bot app, for example).
The value is handler properly and is passed as a proxy option in a call to httpc:set_options([{proxy, {{ProxyAddress, PortNumber}, ["localhost"]}}]). I set the corporate proxy without setting my credentials in the url (in the format of http://[username]:[password]@proxyurl:[proxy_port]. As per my experience Elixir/Erlang does not make use of proxy without credentials attached, so my suspect is that ssl:connect fails with {error, timeout} because it fails to reach the url through the proxy without credentials (something like that). In due time I will try to provide my credentials in the proxy url, but right now I have no idea how to do that because my password now contains @# symbols and at least @ should be escaped as it is used as a separator between [user]:[password]@[proxyurl] in the http_proxy value. All workarounds which I was able to find googling StackOverflow didn't work for me.

So, basically, there are two questions:

1. Would the proxy work with credentials specified in the proxy url
2. How can we pass the proxy (and ideally proxy_auth options) to the ssl:connect/4 call as it seems to be the place when the whole sequence breaks.

I mean, probably, we could set the options for some other Erlang module that is used under the hood inside the ssl module and thus come over this issue.

If you have any idea on how to do it, please drop a line with a hind. I would appreciate that. Thanks!

[acconrad]

@dapdizzy do you want to create a PR to address this? been 2 years and wondering if this is still a problem

[dapdizzy]

I’m currently not actively using this project, but it would be nice to fix this if this is not a super hard fix (in terms of effort). I remember I pointed out the exact place in the ssl module where the problem first manifests itself. I’ll try to lo take some time and look into the problem and if I can help, it’ll be nice to provide a PR to fix this. вт, 25 февр. 2020 г. в 6:48, Adam Conrad <[email protected]>:

…

@dapdizzy <https://github.com/dapdizzy> do you want to create a PR to address this? been 2 years and wondering if this is still a problem — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94?email_source=notifications&email_token=AA7BK6HRIBXV64MSZRIWCWLRESIHZA5CNFSM4CZ4M56KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM2OJ7Y#issuecomment-590669055>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7BK6CS3AUDYVF37HCGA4LRESIHZANCNFSM4CZ4M56A> .