Discovery and bridge access fail with 501 when using a HTTP proxy #171

benbucksch · 2020-05-22T12:54:04Z

Reproduction

Set environment variable https_proxy=myproxy:1234, whereas myproxy:1234 is a Squid proxy. This is a global user config, applying to the entire user session, set e.g. (among others) by the GNOME settings tool for the GNOME browser.
Run hue.discovery.nupnpSearch();
Manually configure the bridge IP address and run hue.api.createLocal(hueBridgeLocalIPAddress).connect();

Actual result

Both calls (step 2 and step 3) return with error message:

Request failed with status code 501

The HTTP proxy is used for contacting the local Hue bridge. That is likely to fail.
The HTTP proxy request used is invalid. The protocol implementation is incorrect. This is a TLS request, so the HTTP CONNECT command is required, but it's not used, leading to the error.

Expected result

If you want to use HTTPS or TLS request via HTTP proxy, the HTTP CONNECT protocol is required.
The HTTP proxy is not used for contacting the Hue bridge, because the bridge is local. The HTTP proxy is for the Internet.
It may be helpful to use the HTTP proxy for contacting the discovery.meethue.com server, but given that the Hue bridge contacts the server without proxy, it's actually more privacy-preserving not to use the proxy, to avoid that the Hue server can associate the proxy IP address and real IP address. So, even in this case, I recommend to ignore the HTTP proxy.

It's better to completely un-support and ignore proxies than doing an invalid request that will fail.

In short: Don't use the HTTP proxy at all. It makes no sense for this API.

The text was updated successfully, but these errors were encountered:

benbucksch · 2020-05-22T13:05:52Z

Here's the response:

response: {
    status: 501,
    statusText: 'Not Implemented',
    headers: {
      server: 'squid/3.5.12',
      'mime-version': '1.0',
      date: 'Fri, 22 May 2020 12:39:29 GMT',
      'content-type': 'text/html;charset=utf-8',
      'content-length': '3549',
      'x-squid-error': 'ERR_UNSUP_REQ 0',
      vary: 'Accept-Language',
      'content-language': 'en',
      'x-cache': 'MISS from myhttpproxy',
      'x-cache-lookup': 'NONE from myhttpproxy:3100',
      connection: 'close'
    },
    config: {
      url: 'https://myhuebridge:443/api/config',
      method: 'get',
      headers: [Object],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      adapter: [Function: httpAdapter],
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      validateStatus: [Function: validateStatus],
      httpsAgent: [Agent],
      data: undefined
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket: [Socket],
      _header: 'GET https://myhuebridge:443/api/config HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'User-Agent: axios/0.19.2\r\n' +
        'host: myhuebridge:443\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      path: 'https://myhuebridge:443/api/config',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      _redirectable: [Writable],
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: <htmlPage>
  },

with HTML error page:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta type="copyright" content="Copyright (C) 1996-2015 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>ERROR: The requested URL could not be retrieved</title>
<style type="text/css"><!-- 
 /*
 * Copyright (C) 1996-2015 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.
 */

/*
 Stylesheet for Squid Error pages
 Adapted from design by Free CSS Templates
 http://www.freecsstemplates.org
 Released for free under a Creative Commons Attribution 2.5 License
*/

/* Page basics */
* {
\tfont-family: verdana, sans-serif;
}

html body {
\tmargin: 0;
\tpadding: 0;
\tbackground: #efefef;
\tfont-size: 12px;
\tcolor: #1e1e1e;
}

/* Page displayed title area */
#titles {
\tmargin-left: 15px;
\tpadding: 10px;
\tpadding-left: 100px;
      "\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\n" +
}

/* initial title */
#titles h1 {
\tcolor: #000000;
}
#titles h2 {
\tcolor: #000000;
}

/* special event: FTP success page titles */
#titles ftpsuccess {
\tbackground-color:#00ff00;
\twidth:100%;
}

/* Page displayed body content area */
#content {
\tpadding: 10px;
\tbackground: #ffffff;
}

/* General text */
p {
}

/* error brief description */
#error p {
}

/* some data which may have caused the problem */
#data {
}

/* the error message received from the system or other software */
#sysmsg {
}

pre {
    font-family:sans-serif;
}

/* special event: FTP / Gopher directory listing */
#dirmsg {
    font-family: courier;
    color: black;
    font-size: 10pt;
}
#dirlisting {
    margin-left: 2%;
    margin-right: 2%;
}
#dirlisting tr.entry td.icon,td.filename,td.size,td.date {
    border-bottom: groove;
}
#dirlisting td.size {
    width: 50px;
    text-align: right;
    padding-right: 5px;
}

/* horizontal lines */
hr {
margin: 0;
}

/* page displayed footer area */
#footer {
\tfont-size: 9px;
\tpadding-left: 10px;
}


body
:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }
:lang(he) { direction: rtl; }
 --></style>
</head><body id=ERR_UNSUP_REQ>
<div id="titles">
<h1>ERROR</h1>
<h2>The requested URL could not be retrieved</h2>
</div>
<hr>

<div id="content">
<p>The following error was encountered while trying to retrieve the URL: <a href="https://myhuebridge/api/config">https://myhuebridge/api/config</a></p>

<blockquote id="error">
<p><b>Unsupported Request Method and Protocol</b></p>
</blockquote>

<p>Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.</p>

<p>Your cache administrator is <a href="mailto:webmaster?subject=CacheErrorInfo%20-%20ERR_UNSUP_REQ&amp;body=CacheHost%3A%20myhttpproxy%0D%0AErrPage%3A%20ERR_UNSUP_REQ%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Fri,%2022%20May%202020%2012%3A39%3A29%20GMT%0D%0A%0D%0AClientIP%3A%2010.1.1.218%0D%0A%0D%0AHTTP%20Request%3A%0D%0AGET%20%2Fapi%2Fconfig%20HTTP%2F1.1%0AAccept%3A%20application%2Fjson,%20text%2Fplain,%20*%2F*%0D%0AUser-Agent%3A%20axios%2F0.19.2%0D%0AConnection%3A%20close%0D%0AHost%3A%20myhuebridge%0D%0A%0D%0A%0D%0A">webmaster</a>.</p>
<br>
</div>

<hr>
<div id="footer">
<p>Generated Fri, 22 May 2020 12:39:29 GMT by myhttpproxy (squid/3.5.12)</p>
<!-- ERR_UNSUP_REQ -->
</div>
</body></html>

benbucksch · 2020-05-22T13:09:53Z

axios is just broken:

But as explained, even if the HTTP proxy request would work, it would still be wrong to use the proxy.

benbucksch · 2020-05-22T14:05:40Z

Fixed in PR #173

peter-murray · 2020-05-31T14:14:39Z

Thanks for all the details and hard work on this.

The only scenario that I can think of currently that needs to be accessed here is the remote API use of this library. In this case you will be establishing a connection to the internet as it is the Remote API in the cloud provided by meethue.com.

I have been considering the removal of axios as it is starting to pull in more library dependencies that are just not necessary for this API. I have the beginnings of this work as an experiment in a branch that might make sense to complete and just remove axios altogether.

Do you have the ability to test the remote API usage with your code changes? I don't have a proxy set up currently to test with, but can probably get to validating this later in the week when I get some time to configure one for testing.

benbucksch · 2020-05-31T17:29:51Z

establishing a connection to the internet as it is the Remote API in the cloud provided by meethue.com.

Yes, I was thinking of that. See Expected result point 3.

But... How is the Hue bridge connecting to meethue.com ? I don't think it can use a proxy, does it? So, we already know for sure that the connection to meethue.com is possible without proxy. At least from the same network as the Hue bridge is on.

That leaves only the case where 1) I control the lights in my house from somewhere outside of my house and 2) that other place requires a proxy to connect to the Internet. I don't think there are going to be many places where both of this is true at the same time. Theorectically possible, yes, but rather unlikely.

The problem is that, right now, all users with a proxy are failing. Even if they are at home where the Hue lights are, and even if the proxy is optional. It's failing 100% of the time when a proxy environment variable is defined, as far as I could discern. Even if the proxy is working perfectly fine, it's still failing.

My proxy is for privacy purposes, and if I connect directly (from Hue hub) and via proxy (from this API) to the same account, Philips can connect the two IP addresses, which is exactly what I don't want.

benbucksch · 2020-05-31T17:35:27Z

I don't have a proxy set up currently to test with

The good news is that the old code failed 100% of the time, so you can't actually break much :)

peter-murray · 2020-05-31T20:25:34Z

Unfortunately I have to validate all the usage scenarios, so in the use case you have, yes, entirely, I understand the failure.

Unfortunately there are other usage scenarios like the remote API access (api.meethue.com) and usage inthe browser that I need to validate this all against before merging into the release branches.

I will set up a simple proxy later this week to run my test suites against. Once again, thank you for the contribution and bringing this to my attention 😄

peter-murray · 2020-05-31T20:28:16Z

Is this a complete blocker for you currently, as you have a work around (as per the PR that you submitted)? Just asking so that I can prioritise things properly.

benbucksch · 2020-06-18T22:10:24Z

This was completely preventing me from using the lib.
Obviously, I have fixed it, and I use my version of the lib. So, I fixed it for my project, with a fork.
However, anybody with a proxy (set using the GNOME UI settings to set a proxy, or similar means) is going to be blocked by this. The lib, and all apps that depend on it, will simply not work. This affects users, not just devs.

peter-murray · 2020-06-22T10:51:27Z

You are the first to have reported this, hence I don't think it is that common in practice (or there are a lot of quiet people out there facing it, but not reporting it anywhere).

I am looking at the possibility of removing axios with a fetch based versions for the http communications, as axios has grown in size and dependencies it has pulled in.

The use case of this library for users covers more than just Node.js server side usage, I have a number of users the embed it in the browser, which has a number of complications as well. It is taking me much longer than I would have liked to review the impact of the changes you have provided to mutliple environments (as well as having to set up proxying servers to test all of this).

As long as you have a solution for now, I will take some time to review all the possible implications and solutions available here.

peter-murray · 2020-11-05T18:03:29Z

Hi @benbucksch 👋 , I have just built up a branch with passing tests and the node-fetch library as the main branch (along with some improvements to the Type definitions for the model objects).

Are you in a position to test the main branch resolves your proxy issues? Do you require an actual npm release to test this? If so let me know and I will cut a pre-release for you.

I have some further testing to do on this for the remote and browser use cases but after that and your confirmation that it works for you, it will be published as a release.

jclerc · 2020-12-03T19:36:55Z

Hi!

I have the same issue as @benbucksch, especially when trying to connect to a local bridge but through an http proxy.

I didn't find a way to make it work using the 4.0.9 release, but I switched to the main branch instead and used global-agent as well, it does now work!

If anyone else stumble across this issue, here is a quick proxy setup:

require('global-agent').bootstrap();

global.GLOBAL_AGENT.HTTP_PROXY = 'http://127.0.0.1:1080'; // can also use environment variable

const host = '192.168.1.XXX'; // has to be your bridge's ip
const user = 'XXX'; // and your user key

require('node-hue-api').v3.api.createLocal(host).connect(user).then((api) => {
  api.lights.getAll().then(lights => console.log(`found ${lights.length} lights!`));
});

node test.js
found 11 lights!

🎉

peter-murray · 2020-12-08T14:23:03Z

Thanks, I will be cutting a new 5.x release soon with this fix in it

benbucksch changed the title ~~Discovery and bridge access fail with 501 when using local HTTP proxy~~ Discovery and bridge access fail with 501 when using a HTTP proxy May 22, 2020

benbucksch linked a pull request May 22, 2020 that will close this issue

Disable HTTP proxy #173

Open

benbucksch mentioned this issue Aug 14, 2020

getCachedState() returns null, failing the requests #172

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Discovery and bridge access fail with 501 when using a HTTP proxy #171

Discovery and bridge access fail with 501 when using a HTTP proxy #171

benbucksch commented May 22, 2020 •

edited

Loading

benbucksch commented May 22, 2020

benbucksch commented May 22, 2020

benbucksch commented May 22, 2020

peter-murray commented May 31, 2020

benbucksch commented May 31, 2020 •

edited

Loading

benbucksch commented May 31, 2020

peter-murray commented May 31, 2020 •

edited

Loading

peter-murray commented May 31, 2020

benbucksch commented Jun 18, 2020 •

edited

Loading

peter-murray commented Jun 22, 2020

peter-murray commented Nov 5, 2020

jclerc commented Dec 3, 2020

peter-murray commented Dec 8, 2020

Discovery and bridge access fail with 501 when using a HTTP proxy #171

Discovery and bridge access fail with 501 when using a HTTP proxy #171

Comments

benbucksch commented May 22, 2020 • edited Loading

Reproduction

Actual result

Expected result

benbucksch commented May 22, 2020

benbucksch commented May 22, 2020

benbucksch commented May 22, 2020

peter-murray commented May 31, 2020

benbucksch commented May 31, 2020 • edited Loading

benbucksch commented May 31, 2020

peter-murray commented May 31, 2020 • edited Loading

peter-murray commented May 31, 2020

benbucksch commented Jun 18, 2020 • edited Loading

peter-murray commented Jun 22, 2020

peter-murray commented Nov 5, 2020

jclerc commented Dec 3, 2020

peter-murray commented Dec 8, 2020

benbucksch commented May 22, 2020 •

edited

Loading

benbucksch commented May 31, 2020 •

edited

Loading

peter-murray commented May 31, 2020 •

edited

Loading

benbucksch commented Jun 18, 2020 •

edited

Loading