Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add validation of uri before saving DatasetURI objects #24

Open
2 of 3 tasks
mortenwh opened this issue Jul 24, 2018 · 1 comment
Open
2 of 3 tasks

Add validation of uri before saving DatasetURI objects #24

mortenwh opened this issue Jul 24, 2018 · 1 comment

Comments

@mortenwh
Copy link
Contributor

mortenwh commented Jul 24, 2018 •
edited
Loading

The URI generic syntax consists of a hierarchical sequence of five components:

URI = scheme:[//authority]path[?query][#fragment]

DatasetURI.uri is presently not validated. This could be a big problem if a lot of data is wrongly ingested. We should add some validation in the get_or_create method of this model:

  • Check that the generic uri syntax is followed

  • Check that the uri points to an actual resource

  • Correct any wrong uris already added to the database - perhaps using migrations?
mortenwh added a commit that referenced this issue Jul 25, 2018
@mortenwh
#24: Added comment in the code
0c91dd5
@mortenwh
Copy link
Contributor Author

mortenwh commented Oct 24, 2018 •
edited
Loading

The two first tasks are done in #38

mortenwh added a commit that referenced this issue Feb 8, 2019
@mortenwh
#24: added validation to DatasetURI's save method, and changed valida…
862e721 
…tion function to only raise exception if the resource is not available
mortenwh added a commit that referenced this issue Feb 14, 2019
@mortenwh
#24: force explicit import of utility functions and classes. Otherwis…
9b05db2 
…e, the __init__ module causes import errors due to recursive imports
mortenwh added a commit that referenced this issue Feb 14, 2019
@mortenwh
#24: accomodate updated function utils.utils.test_validate_uri, test …
53a75d2 
…validate_uri, and update unit tests that use the function to accomodate the new functionality.
mortenwh added a commit that referenced this issue Feb 14, 2019
@mortenwh
#24: modified comment in DatasetURI's save method
ee718be
@mortenwh mortenwh mentioned this issue Feb 14, 2019
Merged
mortenwh added a commit that referenced this issue Feb 14, 2019
@mortenwh
#24: mock request method so it returns object with status 200
ba43645
mortenwh added a commit that referenced this issue Feb 23, 2019
@mortenwh
#40: after merge with issue #24 - do not raise error in the crawler
6dbbf21
mortenwh added a commit that referenced this issue Feb 24, 2019
@mortenwh
#40 and #24: I discovered that a valid uri to an opendap stream retur…
8b5ed10 
…ned html 400 error although netCDF4.Dataset works fine. If request.status!=200, I therefore try to open with netCDF4.Dataset. If that works, no error is raised..
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: after merge with issue #24 - do not raise error in the crawler
6154d81
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40 and #24: I discovered that a valid uri to an opendap stream retur…
86ae519 
…ned html 400 error although netCDF4.Dataset works fine. If request.status!=200, I therefore try to open with netCDF4.Dataset. If that works, no error is raised..
@opsdep opsdep mentioned this issue Sep 25, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mortenwh