JSON Response parsing error at EnvelopValidation #277
Comments
|
Any news about the solution of the previously mentioned issue? Our server is still going through the usual problem - trying to convert envelop using Geometryservice and getting the same error message over and over again. Every time I check the URL to the geometryservice it is a valid JSON... |
|
is the service you are trying to harvest available publicly so we can take a look. it would be good to have the url. also not sure if the geoportal server can access the task url, if you try the task url (http://tasks.arcgisonline.com/ArcGIS/rest/services/Geometry/GeometryServer/project?f=json&inSR=2039&outSR=4326&geometries=%7B%22geometryType%22%3A%22esriGeometryEnvelope%22%2C%22geometries%22%3A%5B%7B%22xmin%22%3A183476.10500000045%2C%22ymin%22%3A636683.0950000007%2C%22xmax%22%3A191879.34999999963%2C%22ymax%22%3A648499.5600000005%7D%5D%7D) on the server geoportal is running, does it work for you? |
|
OK, I prepared some log-file data for you. From the GTP logfile you can see what servers are being harvested - the servers has a wide range of geographic compositions (USA, Japan, PRC, Brazil, Middle-East, etc), so there are a variety of projections in the services. I also included a text file showing the URL of registered service layer and the printout of the layer metadata from geoportal - you can see that service layers from US and Canada ended up in Africa. Also, there are two dataset from a Russian server from where services of similar locations were either registered correctly or ended up in Singapore. Lastly I deployed a test servlet on the hosting Tomcat, connecting to the geometry service on the task.arcgisonline.com (the one you sent over previously). response received fine, first character is indeed a {. This is a link for a public onedrive folder containing all infromation you may need (logfile, metadata prinotus, screenshot of sucessful servlet connection). https://1drv.ms/f/s!Alm4fNYZo5C0gcR1I4km9S9PTOIwgA gpt.2017-07-21.zip But it looks like i managed to upload these files here... Let me know if you need any further details... Tamas |
|
I tried quite a few ags sites from the log file, unfortunately I was unable to reproduce the issue you are having, also for those in GEOPORTAL_ERROR_EXPLANATIONS.txt, the coordinate I got back is different from yours. e.g. on the first one titled "PLSS Boundaries_9K" I got the following back: ows:WGS84BoundingBox I wonder if it is related to system environment/network configuration, do you have more info such as system environment, version of geoportal? customizations, etc. also you might try tools such as fiddler to check the network communication to see if the requests/response looks correct it's also possible to add some log statement to ServiceInfo.java after line 638 (String response = http.readResponseAsCharacters();) to log the url and response LOGGER.finest(String.format("%s => %s", projectionUrl, response)); |
|
@zguo I cannot check the PLSS Boundaries_9K response anymore as this dataset is not available publicly anymore - we just used this for test anyway. But the response you got back is right anyway - the one I included in the GEOPORTAL_ERROR_EXPLANATIONS.txt is completely wrong. The system is composed as follows: I will add the logger line to the deployed webapp's code as soon as I get the chance and I will let you know what was recorded. |
|
Finally I got some time to rebuild geoportal from source along with the new line for Logger (should this line be left in for the next release perhaps?) and after a few failed builds I managed to have the new WAR file deployed on the server. The result is simple - when there is a failure of parsing the JSON response is purely because the JSONparser/JSONObject receives a completely empty string! As the logging was set to FINEST I save the hassle of posting the whole log file - which went up for several hundred megabytes pretty quickly, so the extracts are as follows:
...and another one...
...and one more...
Just to be on the safe side we deployed the similar architecture on a Windows VM - same issues were occurring. Recorded a Network Monitor session too, but that wasn't too informative - I can provided that too if you need it - just let me know. To be honest I am out of ideas what could be the issue here - I assume the tasks.arcgisonline.com is not being throttled. We do not currently have ArcGIS server licence, only ArcGIS Online access, hence we are forced to use tasks.arcgisonline.com for Geometry services. Well, this is pretty much it. Thanks. Tamas |
|
...not all envelop validation failed, these ones were ok: ...and this one was fine too...
|
|
The log is helpful, we thought might be either the request url or the response. |
|
@zguo well, that is also possible though if we need the printout from the JSON responses as above then I need to redeploy the server again, which takes a bit of a time and I am pretty short on that these days. However, in the meantime I can prepare a list summarising what where those services where the envelop validation failed and what are those where it went through fine. I can also find an arcgis server's services in the current repository where services already failed the envelop validation and try to re-harvest it again - see what will happen. Though I must note that I think these services are re-harvested every fortnight and I did see in the logfile that envelop validation keeps failing, so probably they will fail again - but will confirm that later on. |
|
I tried to load test a few of your requests (each ~100 times concurrently), I got a few of failed results even though most time is a success for the same request, thus wondering if it is something related to the load to the geometry service. for the harvesting test, I think one quick way it is to find a site that is having the issue from the existing log, then try re-harvesting for that site a few times more to see if the same issue happens again with exactly same request. |
|
@zguo Ok, I will try to do that later on today or so. Yes, it seems that this issue is related to the service rather the geoportal itself (but let us confirm that with the agreed checks) - but then it might be a good idea to build in some sort of defensive mechanism in the next release, for example upon failed envelop-validation the system should log the type of HTTP error code it receives (if any as ArcGIS tend to send empty HTTP 200 when error occurs) then the thread should get delayed a second or something and re-try it again for a few times before moving on to the next resource in the queue...the number of re-tries could be perhaps configured from gpt.xml - or switched it off completely. At the end of the day with the advent of cloud systems more and more users relies solely on ArcGIS online, meaning that they will not have access to an in-house ArcGIS server instance for geometry-service. Again - if this issue is related to the service... |
|
I can also try to point the geometry service to another location temporarily - see what the results become then... |
|
@zguo Well, I managed to break our Lucene index, so I had to start rebuilding the whole system (I could not get the index rebuilt properly for some reason) and I think I found the core of the problem. It seems that we were trying to index a large number of resources simultaneously and this might have kept triggering a security countermeasure of the ArcGIS Online geometry service. Since I started repopulating the server I have been adding services to be harvested one-by-one then starting a manual synchronization rather in batches with periodic synch - so far we have not received any empty JSON response from the geometry service. I already managed to add 14 services and harvest them successfully - had no issues whatsoever. I think the real problem for us can happen again when geoportal will start re-harvesting large set of resources on a periodic basis - but I think that can be managed by spreading them out evenly throughout a given time period. Anyway, if I see this issue happening again I will do a more detailed investigations regarding ArcGIS Online geometry service availability - but at this stage it seems we are fine, thanks for the help. |
I am keep getting errors when harvested results are being reprojected using the preconfigured Geometry-service.
I am using the "http://tasks.arcgisonline.com/ArcGIS/rest/services/Geometry/GeometryServer/" GeometryServer. The GTP log file is full of the following errors:
If I call the URL from the exception log it is working fine, the JSON response seems correct. The Tomcat Connector is configured for UTF-8 encoding. If I let the harvest to begin with the "Automatically approve newly acquired resources" tab ticked the resource will get registered without re-projection and the envelopes are all around the world - but not at the right locations.
Any help will be very much appreciated!
Tamas
The text was updated successfully, but these errors were encountered: