Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add scheduled indexing from OpenGeoMetadata #1023

Merged
merged 2 commits into from
Mar 1, 2024
Merged

Add scheduled indexing from OpenGeoMetadata #1023

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
96cc0d9
Update to GeoCombine v0.9
thatbudakguy Feb 29, 2024
00032fd
Add scheduled tasks to pull and index OpenGeoMetadata records
thatbudakguy Feb 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ gem 'twitter-typeahead-rails'
gem 'blacklight_range_limit', '~> 7.0'
gem 'redis', '~> 5.0'
gem 'geo_monitor', '~> 0.7', github: 'geoblacklight/geo_monitor'
gem 'geo_combine', '>= 0.9' # For OpenGeoMetadata indexing
gem 'sidekiq', '~> 7.0'
gem 'whenever', require: false
gem 'bootstrap', '~> 4.0'
Expand Down
6 changes: 5 additions & 1 deletion Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,13 +227,16 @@ GEM
faraday-net_http_persistent (2.1.0)
faraday (~> 2.5)
net-http-persistent (~> 4.0)
faraday-retry (2.2.0)
faraday (~> 2.0)
ffi (1.16.3)
ffi-compiler (1.0.1)
ffi (>= 1.0.0)
rake
geo_combine (0.8.0)
geo_combine (0.9.0)
activesupport
faraday-net_http_persistent (~> 2.0)
faraday-retry (~> 2.2)
git
json-schema
nokogiri
Expand Down Expand Up @@ -573,6 +576,7 @@ DEPENDENCIES
dlss-capistrano
factory_bot_rails (~> 6.2.0)
faraday (~> 2.0)
geo_combine (>= 0.9)
geo_monitor (~> 0.7)!
geoblacklight (~> 3.8)
honeybadger
Expand Down
5 changes: 5 additions & 0 deletions config/schedule.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,8 @@
every 1.day, at: '4:04 am', roles: %i[whenevs] do
rake 'rake earthworks:prune_old_search_data[14]'
end

every :tuesday, at: '5:04 am', roles: %i[whenevs] do
rake 'rake earthworks:opengeometadata:pull'
rake 'rake earthworks:opengeometadata:index'
end
21 changes: 3 additions & 18 deletions lib/tasks/earthworks.rake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,35 +114,20 @@ namespace :earthworks do
desc 'Initialize OpenGeoMetadata repositories'
task :clone do
harvester = Earthworks::Harvester.new(ogm_repos: Settings.OGM_REPOS)
total = harvester.clone_all
puts "Cloned #{total} repositories"
harvester.clone_all
end

desc 'Fetch updated OpenGeoMetadata records for indexing'
task :pull do
harvester = Earthworks::Harvester.new(ogm_repos: Settings.OGM_REPOS)
total = harvester.pull_all
puts "Updated #{total} repositories"
harvester.pull_all
end

desc 'Index OpenGeoMetadata repositories'
task :index do
harvester = Earthworks::Harvester.new(ogm_repos: Settings.OGM_REPOS)
indexer = GeoCombine::Indexer.new
puts "Indexing #{harvester.ogm_path} into #{indexer.solr_url}"
total = indexer.index(harvester.docs_to_index)
puts "Indexed #{total} documents"
end

desc 'Index content from GeoBlacklight sites'
task :harvest_geo_blacklight do
GeoCombine::GeoBlacklightHarvester.configure do
Settings.GEO_BLACKLIGHT_HARVEST_SITES.to_h
end

Settings.GEO_BLACKLIGHT_HARVEST_SITES.to_h.keys.each do |site_key|
Rake::Task['geocombine:geoblacklight_harvester:index'].invoke(site_key)
end
indexer.index(harvester.docs_to_index)
end
end

Expand Down
Loading