In addition to URLs, also bring in full-text from Sinequa servers

[code-geek]

Description

When we bring in URLs right now, we just get the URL and title. We also want to store the full-text in the database (but not show it in the table necessarily). This will allow us to track when fulltext for pages change, and also use this data for LLM purposes.

Implementation Considerations

• Needs a new field in the CandidateURL model
• Might need to preserve the last one so we can see deltas
• Perhaps some sort of hashing could help us (MD5)

Deliverable

• Notification when there are new URLs, new titles, new text, and a clear way to tell

Dependencies

No response

[CarsonDavis]

We will probably need to use the sql endpoint from sinequa. Documentation can be found at this link: https://doc.sinequa.com/en.sinequa-es.v11/Content/en.sinequa-es.devDoc.webservice.rest-search.html#engine-sql

[CarsonDavis]

Existing code using this endpoint can be found in the following files:

• COSMOS/config_generation/api.py

  Line 55 in 42bce45

  url = f"{self.base_url}/api/v1/engine.sql"

• COSMOS/config_generation/minimum_api.py

  Line 60 in 42bce45

  url = f"{self.base_url}/api/v1/engine.sql"

[CarsonDavis]

You may need a token from the server that includes SQL engine access. You can search for tokens in the admin console, and then you make a new token, and you give it permissions.