Add XLoader image to run in it's own container, separate from the CKAN UI container #76
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #66
This PR attempts to create an XLoader image to be used as a replacement for DataPusher.
However it currently has a problem in that the callback to the ckan container uses the resource data URL. This means if the URL is something like
http://localhost:5000/dataset/testset20/resource/26004e4f-d13c-49f3-b370-bc38806050fa
XLoader will try to connect to the ckan container running on localhost:5000 which is the xloader container rather than the ckan containerThere needs to be a modification to the XLoader extension itself to allow similar environment variable to DataPusher ie:
ckan.datapusher.callback_url_base
. This would allow control of over what URL xloader would use to communicate back to the ckan containerIt seems these are currently the only xloader configuration parameters available:
ckanext.xloader.jobs_db.uri
ckanext.xloader.api_token
ckanext.xloader.formats
ckanext.xloader.max_content_length
ckanext.xloader.use_type_guessing
ckanext.xloader.strict_type_guessing
ckanext.xloader.max_type_guessing_length
ckanext.xloader.parse_dates_dayfirst
ckanext.xloader.parse_dates_yearfirst
ckanext.xloader.job_timeout
ckanext.xloader.ignore_hash
ckanext.xloader.max_excerpt_lines
ckanext.xloader.ssl_verify
ckanext.xloader.clean_datastore_tables