Update the keyphrase service to fallback to requesting content from websites if crawl service is unavailable

[ashley-evans]

Value Added

Ensures the keyphrase service can continue to function even if the crawl service is unavailable

Description

Currently the keyphrase service is temporally coupled to the crawl service, meaning that any keyphrase analysis cannot be performed without first receiving the crawled HTML content from the crawl service.

The keyphrase service should be updated to request the content directly from a website if the content cannot be received from the keyphrase service for whatever reason.

The content should then be parsed and stored in the same fashion as the cached content from the crawl service.

Acceptance Criteria

AC01

• Update the keyphrase service to request page content from websites directly if the content cannot be received from the crawl service for whatever reason.

AC02

• The keyphrase service's functionality should be otherwise unaffected: The content should be still stored in the parsed content S3 bucket