Skip to content

Latest commit

 

History

History
executable file
·
187 lines (156 loc) · 8.92 KB

README.md

File metadata and controls

executable file
·
187 lines (156 loc) · 8.92 KB
Raw

Archives of Nethys Search Engine

A configurable, generic search UI for Archives of Nethys Built with Elastic App Search's Reference UI.

Updating configuration

The project can be configured via a JSON config file.

You can easily control things like...

  • The Engine the UI runs against
  • Which fields are displayed
  • The filters that are used

If you would like to make configuration changes, there is no need to regenerate this app from your App Search Dashboard!

You can simply open up the engine.json file, update the options, and then restart this app.

Configuration options

The following is a complete list of options available for configuration in engine.json.

option value type required/optional source
engineName String required Found in your App Search Dashboard.
endpointBase String required* (*) Elastic Enterprise Search deployment URL, example: "http://127.0.0.1:3002". Not required if using App Search on swiftype.com.
hostIdentifier String required* (*) Only required if using App Search on swiftype.com.
searchKey String required Found in your App Search Dashboard.
searchFields Array[String] required A list of fields that will be searched with your search term.
resultFields Array[String] required A list of fields that will be displayed within your results.
querySuggestFields Array[String] optional A list of fields that will be searched and displayed as query suggestions.
titleField String optional The field to display as the title in results.
urlField String optional A field with a url to use as a link in results.
sortFields Array[String] optional A list of fields that will be used for sort options.
facets Array[String] optional A list of fields that will be available as "facet" filters. Read more about facets within the App Search documentation.

Deploy

npm run deploy

Where'd the data come from?

Originally, the data came from a scrape of the Archives of Nethys website, for Pathfinder - 2nd Edition. I scraped it using the Elastic App Search Crawler. When I scraped it, the Crawler was still in "beta", so I did a little extra cleanup to get the data formatted exactly as I wanted it.

First, I ran the crawler in local mode, with a config like:

max_indexed_links_count: 1000
max_extracted_links_count: 10000
max_url_length: 70
max_crawl_depth: 10
domain_allowlist:
  - https://2e.aonprd.com
seed_urls:
  - https://2e.aonprd.com/Actions.aspx
  - https://2e.aonprd.com/Afflictions.aspx
  - https://2e.aonprd.com/Ancestries.aspx
  - https://2e.aonprd.com/Archetypes.aspx
  - https://2e.aonprd.com/Backgrounds.aspx
  - https://2e.aonprd.com/Classes.aspx
  - https://2e.aonprd.com/Conditions.aspx
  - https://2e.aonprd.com/Creatures.aspx
  - https://2e.aonprd.com/Equipment.aspx?All=true
  - https://2e.aonprd.com/Feats.aspx
  - https://2e.aonprd.com/Hazards.aspx
  - https://2e.aonprd.com/Rules.aspx
  - https://2e.aonprd.com/Skills.aspx
  - https://2e.aonprd.com/SpellLists.aspx?Tradition=0
  - https://2e.aonprd.com/Traits.aspx
  - https://2e.aonprd.com/Rules.aspx?ID=1483
  - https://2e.aonprd.com/Rules.aspx?ID=748
  - https://2e.aonprd.com/Rules.aspx?ID=1587
  - https://2e.aonprd.com/Rules.aspx?ID=686
output_sink: file
output_dir: examples/output/aon

Next, I made a new dir:

mkdir examples/output/aon-cleaned/

Then, I cleaned the raw crawl JSON:

require 'json'
require 'nokogiri'

output_dir = 'examples/output/aon-cleaned/'
input_dir = 'examples/output/aon/'
Dir.foreach(input_dir) do |filename|
  next if filename == '.' or filename == '..'
  puts "Parsing #{filename}"
  json = File.read("#{input_dir}#{filename}")
  hsh = JSON.parse(json)
  parsed_data = Nokogiri::HTML.parse(hsh['content'])
  title = parsed_data.title.strip.gsub(' - Archives of Nethys: Pathfinder 2nd Edition Database','')
  categories = title.split(' - ')
  category = categories.size >= 2 ? categories[1] : nil
  sub_category = categories.size >= 3 ? categories[2] :nil
  common_keywords = ["Archives", "Nethys", "Wiki", "Archives of Nethys", "Pathfinder", "Official", "AoN", "AoNPRD", "PRD", "PFSRD", "2E", "2nd Edition"]
  keywords =  parsed_data.at('meta[name=keywords]')['content'].split(', ') - common_keywords
  description = parsed_data.at('meta[name=description]')['content']
  description = Nokogiri::HTML(description)&.text
  body_content = parsed_data.at_css('[id="ctl00_MainContent_DetailedOutput"]')&.text
  body_content = parsed_data.at_css('[id="ctl00_RadDrawer1_Content_MainContent_DetailedOutput"]')&.text unless body_content
  result = {
    :title => title,
    :category => category,
    :sub_category => sub_category,
    :keywords => keywords,
    :description => description,
    :body => body_content,
    :url => hsh['url']
  }
  output_json = JSON.pretty_generate(result)
  File.open("#{output_dir}#{filename}", 'w') { |file| file.write(output_json) }
  puts "Parsed #{filename}"
rescue StandardError => e
  puts "Failed to parse #{filename} because #{e.class}: #{e.message}"
end

Then, I used the App Search Ruby Client (gem install elastic-enterprise-search) to index the cleaned json

require 'elastic-enterprise-search'

host = <host>
key = <private key>

ent_client = Elastic::EnterpriseSearch::Client.new(host: host)
ent_client.app_search.http_auth = key
client = ent_client.app_search
engine_name = 'aon-non-crawl'
documents = []
batch = 1
output_dir = 'examples/output/aon-cleaned/'
batch_ids = {}
Dir.foreach(output_dir) do |filename|
  next if filename == '.' or filename == '..'
  document = JSON.parse(File.read("#{output_dir}#{filename}"))
  document[:id] = document.dup.slice('title', 'category', 'keywords', 'description', 'body').hash.to_s
  unless batch_ids.keys.include?(document[:id])
    documents << document
    batch_ids[document[:id]] = true
  end
  if documents.size == 100
    puts "indexing batch #{batch}"
    client.index_documents(engine_name, documents: documents)
    documents = []
    batch_ids = {}
    batch = batch + 1
  end
end
client.index_documents(engine_name, documents: documents)

If something went wrong, and we wanted to delete the engine contents without deleting the engine itself, we can run:

batch = 0
loop do
  batch += 1
  result = client.list_documents(engine_name)
  ids = result.body['results'].map{|it| it['id']}
  if ids.empty?
    break
  end
  puts "Deleting batch #{batch}"
  client.delete_documents(engine_name, document_ids: ids)
end

License 📗

Apache-2.0 © Elastic