Missing Urls Analysis - Helio_Events_Knowledgebase_Website

[CarsonDavis]

Some Basics

HEK: the Heliophysics Events Knowledgebase. it contains the following two components

• HER: Heliophysics Events Registry
  • database of events that occurred on the sun, such a flare
• HCR: Heliophysics Coverage Registry
  • database of operational sequences
  • records that a telescope was looking at the sun's north polar region at a particular time on a particular day

There are a handful of core website and documentation pages, as well as a few search pages I haven't fully differentiated between. The rest of the pages are either events or filtered event lists.

Supposedly, isolsearch is the main ui tool that users use to search the hek, however there are other search interfaces on the site.

Lockheed and Stanford both have involvement in this project, and there may or may not be relevant stuff hosted on their websites. For example the HER API documentation is located on Stanford's website.

There is a python package called sunpy which lets users interface with the HEK api. It is also hosted on its own page.

URL Stats

Here is the spreadsheet that lists all 80,519 urls we have ever found on the website and contains a preliminary assessment of each url.

• Raw Counts
  • total unique: 80,519
  • new 13,390
  • original 67,131
• Deltas
  • missing from new: 67,128
  • missing from original: 13,389

Discoveries

Multiplying Urls in original connector

• For a single event, the original connector would actually list 3 urls:
  • the event
  • the event with the full parameter
  • the event with :// replaced by %3A%2F%2F
• An example is shown below:
  • https://www.lmsal.com/hek/her?cmd=view-voevent-full&ivorn=ivo://helio-informatics.org/AR171_AlbertoSainzDalda_20100630_215741
  • https://www.lmsal.com/hek/her?cmd=view-voevent-full&ivorn=ivo%3A%2F%2Fhelio-informatics.org%2FAR171_AlbertoSainzDalda_20100630_215741&rev=1
  • https://www.lmsal.com/hek/her?cmd=view-voevent&ivorn=ivo://helio-informatics.org/AR171_AlbertoSainzDalda_20100630_215741

Connector Differences

The differences shown below are between the sde and smd folders, and not necessarily between the two document collections. These differences certainly wouldn't account for the url discrepancies we are seeing.

• title mapping moved to top
• doc type mapping moved to top
• shard index removed

Questionable Settings

• https://www.lmsal.com/hek/her?cmd=home is directly included. However, it is also directly excluded. The exclude lacks a star at the end, so this might just exclude the root and be intentional.
• www.lmsal.com/hek/her?cmd=home&offset=* is excluded separately, which should not be needed
• www.lmsal.com/hek/index.html is directly included. It is also excluded...
• the version is set based on the url, which is meaningless
• What is <SelectionQuery>title &lt;&gt; 'Heliophysics Event Registry'</SelectionQuery> doing?

Site Analysis

What do we definitely want?

• https://www.lmsal.com/hek/
• https://www.lmsal.com/get_aia_data/
• https://www.lmsal.com/heksearch/
• https://www.lmsal.com/hek/her?cmd=contact
• each event
  • it will be important that whatever page this is contains all the right stuff. the movie file link, the comments from annotations, event description, etc
• https://www.lmsal.com/sdodocs/
  • this is a goldmine of informative pdfs, powerpoints, and other docs. we probably need this main page and every link on it

What do we possibly want?

• https://docs.sunpy.org/en/stable/tutorial/acquiring_data/hek.html
  • very useful tutorial on how to use the sunpy extension on top of the helio events knowledgebase
  • this seems mostly useful for using a python plugin to interact with the events database
• https://www.lmsal.com/isolsearch
  • this is the actual interactive search ui for the hek
• http://solar.stanford.edu/hekwiki/ApplicationProgrammingInterface?action=print
  • this is the api documentation for the her... why are we not using this?
• recent events page
  • https://www.lmsal.com/hek/her?cmd=home
  • i think we probably want this page, imagine a user searching for recent solar events. however, we do not want to index the individual event links on this page, as they would be duplicates of the complete event listing
• recommended events
  • https://www.lmsal.com/hek/hcr?cmd=view-recommended-events
  • 63 events considered to be particularly noteworthy
• popular events
  • https://www.lmsal.com/hek/hcr?cmd=view-popular-events
  • these are 217 thousand of the most downloaded events, filterable by instrument. maybe we get the main page, but i definitely don't think we should follow each link

What do we probably NOT want?

• random queries, or example queries, such as those found on the main HEK page:
  • https://www.lmsal.com/isolsearch?hek_query=http://www.lmsal.com/hek/her?cosec=2&cmd=search&type=column&event_type=ar,ce,cj,cw,ef,fi,fe,fa,fl,os&event_region=all&event_coordsys=helioprojective&x1=-5000&x2=5000&y1=-5000&y2=5000&result_limit=200&event_starttime=2011-02-14T00:00:00&event_endtime=2011-02-15T00:00:00
  • https://www.lmsal.com/hek/index.html is a duplicate of https://www.lmsal.com/hek/
• https://www.lmsal.com/hek/her?cmd=home&offset=20
  • stuff like this is a next page listing from the recent events, and we probably don't need it, since it is only important to point users to the existence of a recent events page, not exhaustively show everything considered to be a recent event, separate from the main listing of all the events
  • it should be noted that this offset goes on to huge numbers; i tried 10,000 and was still getting results. i'm guessing it's actually all events ordered by date?
• planned observations
  • https://www.lmsal.com/hek/hcr?cmd=view-planned-events
  • these are observations planned to occur in the next 24 hours. weirdly, it says there are 184 THOUSAND of them. idk if this is accurate or not, but either way, this page will be irrelevant a day after it's indexed, so we would definitely need to think about how or even whether, we would index it

Lingering Questions

• why are we not using an api oriented connector??
• what is the difference between these search urls?
  • https://www.lmsal.com/heksearch/
  • https://www.lmsal.com/isolsearch
  • https://www.lmsal.com/hek/hcr?cmd=search-events
• view-event vs view-voevent