HttpFeed doesn't detect URL unreachable after VM termination

[aledsage]

I have an entity that does:

        httpFeed = HttpFeed.builder()
                .entity(this)
                .period(200)
                .baseUri(rootUri)
                .poll(new HttpPollConfig<Boolean>(ROOT_URL_UP)
                        .onSuccess(HttpValueFunctions.responseCodeEquals(200))
                        .onFailureOrException(Functions.constant(false)))
                .build();

I terminated the aws-ec2:us-east-1 VM behind the entity's back, expecting to see this attribute report false in a timely manner. However, after 15 minutes it still says true. Looking in jstack, I see this long running thread that appears to be stuck:

"brooklyn-execmanager-hqiDYj2X-1018" daemon prio=5 tid=7f9f250de800 nid=0x119eda000 runnable [119ed8000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:251)
at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:209)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:171)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:201)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:155)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:169)
at brooklyn.event.feed.http.HttpPollValue.(HttpPollValue.java:48)
at brooklyn.util.http.HttpTool.execAndConsume(HttpTool.java:275)
at brooklyn.util.http.HttpTool.httpGet(HttpTool.java:246)
at brooklyn.event.feed.http.HttpFeed$1.call(HttpFeed.java:287)
at brooklyn.event.feed.http.HttpFeed$1.call(HttpFeed.java:1)
at brooklyn.event.feed.Poller$PollJob$1.run(Poller.java:53)
at brooklyn.event.feed.Poller$1$1.call(Poller.java:117)
at brooklyn.event.feed.Poller$1$1.call(Poller.java:1)
at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:296)
at brooklyn.util.task.BasicExecutionManager$2$1.call(BasicExecutionManager.java:299)
at brooklyn.util.task.BasicExecutionManager$3.call(BasicExecutionManager.java:352)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)

Also note that going to the URL in a browser takes about 80 seconds before it reports:

Oops! Google Chrome could not connect to ec2-54-227-75-240.compute-1.amazonaws.com:9080

[ahgittin]

I think we need to specify connection timeout and completion timeouts on the stream. Tedious but seems like the right answer, with sensible defaults but possibly configurable longer term.

[aledsage]

Agree with setting httpConnectionParams.getParams().setConnectionTimeout() and setSoTimeout() [1].

Maybe the SoTimeout will be enough ("which is the timeout for waiting for data"), or maybe that's just about opening the socket initially?

Adding timeouts on the steam once we're into calling ByteStreams.copy doesn't look possible, except by calling it in another thread that that can be timed out.

Including a timeout on the higher-level brooklyn task looks feasible (see brooklyn.event.feed.Poller.start(), where it creates and submits the ScheduledTask) but not straight forward. As an aside, I don't like our ScheduledTask code - I think it should look much more like the methods in java.util.concurrent.ScheduledExecutorService rather than it being implicit from the type of the task submitted being a ScheduledTask.

Conclusion: I'll try adding setConnectionTimeout and setSoTimeout, configured in the HttpFeed, and set to 30 seconds in my use-case.

[1] https://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/params/HttpConnectionParams.html#setSoTimeout(int)

[ahgittin]

Maybe the SoTimeout will be enough ("which is the timeout for waiting
for data"), or maybe that's just about opening the socket initially?
Not sure.

Adding timeouts on the steam once we're into calling
|ByteStreams.copy| doesn't look possible, except by calling it in
another thread that that can be timed out.

Worried that might be the case. Worst case we could duplicate
ByteStreams. Although I think it may be possible to register a factory
to be used somehow.

Including a timeout on the higher-level brooklyn task looks feasible
(see |brooklyn.event.feed.Poller.start()|, where it creates and
submits the |ScheduledTask|) but not straight forward. As an aside, I
don't like our |ScheduledTask| code - I think it should look much more
like the methods in |java.util.concurrent.ScheduledExecutorService|
rather than it being implicit from the type of the task submitted
being a |ScheduledTask|.

Timeouts at both levels, with slightly longer on the task, is probably
the way to go. IIRC network timeouts can be surprising.

Agree that ScheduledTask API could be improved!