Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stuck on an issue. hiedr2mosaic.py -> spiceinit -> The server sent an unrecognized response. -> 'FileNotFoundError: [Errno 2] No such file or directory: 'flat_0_1.txt' #437

Open
morrigan-j opened this issue Jul 2, 2024 · 5 comments
Open

Stuck on an issue. hiedr2mosaic.py -> spiceinit -> The server sent an unrecognized response. -> 'FileNotFoundError: [Errno 2] No such file or directory: 'flat_0_1.txt' #437

morrigan-j opened this issue Jul 2, 2024 · 5 comments

Comments

@morrigan-j
Copy link

morrigan-j commented Jul 2, 2024 •
edited
Loading

Hello all,

Apologies if this is an inappropriate use of this, but I've been having this issue for months now and have thus far been unable to resolve it with any solutions to similar issues I've seen raised here.

I'm trying to ultimately make a Digital Elevation Model from HiRISE imagery. However, whenever I use the 'hiedr2mosaic.py' script to download and pre-process the images into a .cub file, it's always ending with the error: 'hiedr2mosaic.py 'FileNotFoundError: [Errno 2] No such file or directory: 'flat_0_1.txt' . As far as I can see, I believe this may be a problem connecting with the web service whenever it's trying to pull Kernal data, but I am frankly unsure. I have attempted to download the full MRO dataset to test this, but thus far have been having storage and 'local' connection issues that have thus far prevented me from completing the entire download.

For a brief overview of the common errors I tend to face:

hi2isis completes just fine.
hical completes just fine.
histitch completes just fine
spiceinit WEB=True seems to work for RED0-RED3, then from RED4onwards -> 'ERROR The server sent an unrecognized response'
spicefit looks to work for RED0 and RED1, then from RED2 onwards -> 'Programmer ERROR Parameter [FROM] is not a filename'
Subsequent errors then include:
'Unable to initialize camera model in Camera Factory',
'PVL Keyword [TargetPosition] does not exist in [Group = Kernals],
'I/O Error Unable to open [ESP_029421_3200_RED0.nonproj.cub]', etc.
before finally ending with: 'FileNotFoundError: [Errno 2] No such file or directory: 'flat_0_1.txt'.

This issue has really stumped me for a while, so I'd appreciate any help anyone could give! (and apologies if its a simply fix that I'm just missing)

I was using ISIS version 8.0.0_RC2 when this particular attempt below was made.

Here's the exact commandline I'm using:

'python hiedr2mosaic.py --download-folder mos_cubes/hirise_example/ http://hirise-pds.lpl.arizona.edu/PDS/EDR/ESP/ORB_029400_029499/ESP_029421_2300/ --web'

and here's a copy paste of the terminal log from spiceinit onwards:

spiceinit WEB=True from=ESP_029421_2300_RED0.histitch.cub
spiceinit WEB=True from=ESP_029421_2300_RED1.histitch.cub
spiceinit WEB=True from=ESP_029421_2300_RED2.histitch.cub
spiceinit WEB=True from=ESP_029421_2300_RED3.histitch.cub
ERROR The server sent an unrecognized response.
spiceinit WEB=True from=ESP_029421_2300_RED4.histitch.cub
ERROR The server sent an unrecognized response.
spiceinit WEB=True from=ESP_029421_2300_RED5.histitch.cub
ERROR The server sent an unrecognized response.
spiceinit WEB=True from=ESP_029421_2300_RED6.histitch.cub
ERROR The server sent an unrecognized response.
spiceinit WEB=True from=ESP_029421_2300_RED7.histitch.cub
ERROR The server sent an unrecognized response.
spiceinit WEB=True from=ESP_029421_2300_RED8.histitch.cub
Waiting for jobs to finish
ERROR The server sent an unrecognized response.
ERROR The server sent an unrecognized response.
ERROR The server sent an unrecognized response.
ERROR The server sent an unrecognized response.
spicefit from= ESP_029421_2300_RED0.histitch.cub
spicefit from= ESP_029421_2300_RED1.histitch.cub
spicefit from= ESP_029421_2300_RED2.histitch.cub
PROGRAMMER ERROR Parameter [FROM] is not a filename.
spicefit from= ESP_029421_2300_RED3.histitch.cub
PROGRAMMER ERROR Parameter [FROM] is not a filename.
spicefit from= ESP_029421_2300_RED4.histitch.cub
PROGRAMMER ERROR Parameter [FROM] is not a filename.
spicefit from= ESP_029421_2300_RED5.histitch.cub
PROGRAMMER ERROR Parameter [FROM] is not a filename.
spicefit from= ESP_029421_2300_RED6.histitch.cub
PROGRAMMER ERROR Parameter [FROM] is not a filename.
spicefit from= ESP_029421_2300_RED7.histitch.cub
PROGRAMMER ERROR Parameter [FROM] is not a filename.
spicefit from= ESP_029421_2300_RED8.histitch.cub
PROGRAMMER ERROR Parameter [FROM] is not a filename.
Waiting for jobs to finish
PROGRAMMER ERROR Parameter [FROM] is not a filename.
PROGRAMMER ERROR Parameter [FROM] is not a filename.
mkdir -p tmp_ESP_029421_2300_RED0.histitch.cub&& cd tmp_ESP_029421_2300_RED0.histitch.cub&& noproj from=../ESP_029421_2300_RED0.histitch.cub match=../ESP_029421_2300_RED5.histitch.cub source= frommatch to=../ESP_029421_2300_RED0.noproj.cub&& cd .. && rm -rf tmp_ESP_029421_2300_RED0.histitch.cub
mkdir -p tmp_ESP_029421_2300_RED1.histitch.cub&& cd tmp_ESP_029421_2300_RED1.histitch.cub&& noproj from=../ESP_029421_2300_RED1.histitch.cub match=../ESP_029421_2300_RED5.histitch.cub source= frommatch to=../ESP_029421_2300_RED1.noproj.cub&& cd .. && rm -rf tmp_ESP_029421_2300_RED1.histitch.cub
mkdir -p tmp_ESP_029421_2300_RED2.histitch.cub&& cd tmp_ESP_029421_2300_RED2.histitch.cub&& noproj from=../ESP_029421_2300_RED2.histitch.cub match=../ESP_029421_2300_RED5.histitch.cub source= frommatch to=../ESP_029421_2300_RED2.noproj.cub&& cd .. && rm -rf tmp_ESP_029421_2300_RED2.histitch.cub
mkdir -p tmp_ESP_029421_2300_RED3.histitch.cub&& cd tmp_ESP_029421_2300_RED3.histitch.cub&& noproj from=../ESP_029421_2300_RED3.histitch.cub match=../ESP_029421_2300_RED5.histitch.cub source= frommatch to=../ESP_029421_2300_RED3.noproj.cub&& cd .. && rm -rf tmp_ESP_029421_2300_RED3.histitch.cub
ERROR Unable to initialize camera model in Camera Factory.
ERROR PVL Keyword [TargetPosition] does not exist in [Group = Kernels].
mkdir -p tmp_ESP_029421_2300_RED4.histitch.cub&& cd tmp_ESP_029421_2300_RED4.histitch.cub&& noproj from=../ESP_029421_2300_RED4.histitch.cub match=../ESP_029421_2300_RED5.histitch.cub source= frommatch to=../ESP_029421_2300_RED4.noproj.cub&& cd .. && rm -rf tmp_ESP_029421_2300_RED4.histitch.cub
ERROR Unable to initialize camera model in Camera Factory.
ERROR PVL Keyword [TargetPosition] does not exist in [Group = Kernels].
mkdir -p tmp_ESP_029421_2300_RED5.histitch.cub&& cd tmp_ESP_029421_2300_RED5.histitch.cub&& noproj from=../ESP_029421_2300_RED5.histitch.cub match=../ESP_029421_2300_RED5.histitch.cub source= frommatch to=../ESP_029421_2300_RED5.noproj.cub&& cd .. && rm -rf tmp_ESP_029421_2300_RED5.histitch.cub
ERROR Unable to initialize camera model in Camera Factory.
ERROR PVL Keyword [TargetPosition] does not exist in [Group = Kernels].
mkdir -p tmp_ESP_029421_2300_RED6.histitch.cub&& cd tmp_ESP_029421_2300_RED6.histitch.cub&& noproj from=../ESP_029421_2300_RED6.histitch.cub match=../ESP_029421_2300_RED5.histitch.cub source= frommatch to=../ESP_029421_2300_RED6.noproj.cub&& cd .. && rm -rf tmp_ESP_029421_2300_RED6.histitch.cub
ERROR Unable to initialize camera model in Camera Factory.
ERROR PVL Keyword [TargetPosition] does not exist in [Group = Kernels].
mkdir -p tmp_ESP_029421_2300_RED7.histitch.cub&& cd tmp_ESP_029421_2300_RED7.histitch.cub&& noproj from=../ESP_029421_2300_RED7.histitch.cub match=../ESP_029421_2300_RED5.histitch.cub source= frommatch to=../ESP_029421_2300_RED7.noproj.cub&& cd .. && rm -rf tmp_ESP_029421_2300_RED7.histitch.cub
ERROR Unable to initialize camera model in Camera Factory.
ERROR PVL Keyword [TargetPosition] does not exist in [Group = Kernels].
mkdir -p tmp_ESP_029421_2300_RED8.histitch.cub&& cd tmp_ESP_029421_2300_RED8.histitch.cub&& noproj from=../ESP_029421_2300_RED8.histitch.cub match=../ESP_029421_2300_RED5.histitch.cub source= frommatch to=../ESP_029421_2300_RED8.noproj.cub&& cd .. && rm -rf tmp_ESP_029421_2300_RED8.histitch.cub
ERROR Unable to initialize camera model in Camera Factory.
ERROR PVL Keyword [TargetPosition] does not exist in [Group = Kernels].
Waiting for jobs to finish
ERROR Unable to initialize camera model in Camera Factory.
ERROR PVL Keyword [TargetPosition] does not exist in [Group = Kernels].
ERROR Unable to initialize camera model in Camera Factory.
ERROR PVL Keyword [TargetPosition] does not exist in [Group = Kernels].
ERROR Unable to initialize camera model in Camera Factory.
ERROR PVL Keyword [TargetPosition] does not exist in [Group = Kernels].
hijitreg from= ESP_029421_2300_RED0.noproj.cub match= ESP_029421_2300_RED1.noproj.cub flatfile= flat_0_1.txt
hijitreg from= ESP_029421_2300_RED1.noproj.cub match= ESP_029421_2300_RED2.noproj.cub flatfile= flat_1_2.txt
hijitreg from= ESP_029421_2300_RED2.noproj.cub match= ESP_029421_2300_RED3.noproj.cub flatfile= flat_2_3.txt
I/O ERROR Unable to open [ESP_029421_2300_RED0.noproj.cub].
hijitreg from= ESP_029421_2300_RED3.noproj.cub match= ESP_029421_2300_RED4.noproj.cub flatfile= flat_3_4.txt
I/O ERROR Unable to open [ESP_029421_2300_RED1.noproj.cub].
hijitreg from= ESP_029421_2300_RED4.noproj.cub match= ESP_029421_2300_RED5.noproj.cub flatfile= flat_4_5.txt
I/O ERROR Unable to open [ESP_029421_2300_RED2.noproj.cub].
hijitreg from= ESP_029421_2300_RED5.noproj.cub match= ESP_029421_2300_RED6.noproj.cub flatfile= flat_5_6.txt
I/O ERROR Unable to open [ESP_029421_2300_RED3.noproj.cub].
hijitreg from= ESP_029421_2300_RED6.noproj.cub match= ESP_029421_2300_RED7.noproj.cub flatfile= flat_6_7.txt
I/O ERROR Unable to open [ESP_029421_2300_RED4.noproj.cub].
hijitreg from= ESP_029421_2300_RED7.noproj.cub match= ESP_029421_2300_RED8.noproj.cub flatfile= flat_7_8.txt
I/O ERROR Unable to open [ESP_029421_2300_RED5.noproj.cub].
Waiting for jobs to finish
I/O ERROR Unable to open [ESP_029421_2300_RED6.noproj.cub].
I/O ERROR Unable to open [ESP_029421_2300_RED7.noproj.cub].
Traceback (most recent call last):
File "/home/[username]/StereoPipeline/bin/hiedr2mosaic.py", line 475, in
sys.exit(main())
File "/home/[username]/StereoPipeline/bin/hiedr2mosaic.py", line 448, in main
averages = hijitreg( noprojed_CCDs, options.threads, options.delete )
File "/home/[username]/StereoPipeline/bin/hiedr2mosaic.py", line 258, in hijitreg
averages[i] = read_flatfile( flat_file )
File "/home/[username]/StereoPipeline/bin/hiedr2mosaic.py", line 90, in read_flatfile
f = open(flat,'r')
FileNotFoundError: [Errno 2] No such file or directory: 'flat_0_1.txt'
@oleg-alexandrov
Copy link
Member

oleg-alexandrov commented Jul 2, 2024 via email

@rbeyer
Copy link
Member

rbeyer commented Jul 2, 2024

@morrigan-j, so it doesn't work for RED0-4. The hiedr2mosaic.py program parallelizes itself, and works on the inputs in parallel. As a consequence, the order in which things are reported to STDOUT is the order in which they happen from all of the processes running, and it just takes a few seconds for the first web responses to come back. So if you count the number of "ERROR The server sent an unrecognized response." occurrences, you'll see that there are nine of them, one for every input. This lack of ability to access the kernel information is the cause of all of the remaining errors. It seems like there is a problem with accessing the USGS's spice server. Sadly, the solution is to download hundreds of GB of MRO data. Hopefully, you can see if you can clear up access to the USGS web spice server (sometimes it is server-side issues and down-time, sometimes it is the connection between your computer and their servers).

@Kelvinrr
Copy link

Kelvinrr commented Jul 2, 2024 •
edited
Loading

This is the second issue where people are having issues using the spice server, but our uptime robots don't show any gaps in the service. @morrigan-j are you able to hit the endpoint? what do you get if you run curl https://astrogeology.usgs.gov/apis/ale/v0.9.1/spiceserver/?

@Kelvinrr
Copy link

Kelvinrr commented Jul 3, 2024

Looking more into this it might be a problem with the redirect on old versions of ISIS, if you can change the spiceinit calls to have url=https://astrogeology.usgs.gov/apis/ale/v0.9.1/spiceserver/, it might solve the issue.

@morrigan-j
Copy link
Author

Apologies all for taking a while to reply, I've had quite a busy few weeks and haven't had much chance to take another look at this.

In that time however, I did manage to get all the ISIS data downloaded, though my folder came to 329GB of data for MRO alone 😬 . Nevertheless, this did work to get me around the issue. I'll see about raising the issue further over on the ISIS3 GitHub, as the underlying issue ultimately does still exist.

@Kelvinrr When I run 'curl https://astrogeology.usgs.gov/apis/ale/v0.9.1/spiceserver/' I get the following response:

{"body":{"status":{"is_app_available":true,"is_cache_available":false,"is_spice_available":true}}}

I'll see about giving updating to a newer version of ISIS a try to see if that resolves the issue when I get a minute.

Thanks everyone for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rbeyer @oleg-alexandrov @Kelvinrr @morrigan-j