Server side search, filter and sort

[ailisp]

Problems

Currently, the search, filter by author or label, sorting by newest or latest are happens in smart contract and frontend. Then the result, such as search, filter and sorting is combined together in the frontend. There is a couple of problems with this approach:

• It performs bad and doesn't scale when many more posts are added in future. Even now we cannot search non top level posts for performance reason.
• The full text fuzzy search isn't robust. It can produce confusing result because goes far for trying fuzzing the search result.
• It is implemented in a hacky way. The code to do combination in frontend is difficult to read and maintain. It is hard to extend to more kinds of filters and sorting in future.

User Story
As a user of DevHub, I want the search, filter and sort experience to be smooth and reliable. They should also work together. As a maintainer of DevHub, I want this part of code is easy to maintain and can be extend for future needs.

Acceptance Criteria

• Implement the indexer from data team's QueryAPI Dashboard, as demoed here: https://www.loom.com/share/77ab9a4acbf34e1aa65961dc70459a22 to avoid maintain an indexer server infra.
• The indexer can do full text search posts. If there is a precise match, the result should be returned. Otherwise, the fuzzy search result is returned.
• The indexer can filter the post by author and label
• The indexer can sort the post by newest and hottest
• The indexer can combine above results.
• Remove frontend combination resullt logics from widgets
• Remove unnecessary smart contract side filter logics from contract

Technical Details

1. The most recent version of QueryAPI widget
2. Search DevHub contract stored Posts, which technically it's from "add_post", "edit_post" transaction content.
3. Query API may or may not work with historical blocks but only new blocks. If not, it means only posts after the indexer launched will be processed and indexed and we'll need to do a one-time processing for existing posts either from historical blocks, or a dump of contract state.

** Spike breakdown **

• Create database schema to store necessary info extract from transactions
• Create indexingLogic.js to populate data and store in database
• Create graphql queries to read (search & filter) from the database
• Call graphql queries from frontend and replace current search & filter logics

[carina-akaia]

I'd like to tackle this ticket in the next 2 weeks as well

[carina-akaia]

There's probably a blocker, since indexer deployment requires an access key for queryapi.dataplatform.near. I actually think that this feature still can be properly implemented entirely on frontend, but it would require at least 2 weeks to achieve a good architectural approach in order to make it well-maintainable.

For now, I'm switching to a P2 task until further updates take place.

[ailisp]

Switched to p2.

There's probably a blocker, since indexer deployment requires an access key for queryapi.dataplatform.near

I'll ask them. Can you send me your public key in tg (my handle is the same as my github)

I actually think that this feature still can be properly implemented entirely on frontend, but it would require at least 2 weeks to achieve a good architectural approach in order to make it well-maintainable.

I think the complexity doesn't overweigh the infra simplification. Implement in indexer would make frontend logic clear and it's a one time solution that scale for more posts & more kinds of filtering.

[carina-akaia]

I'll ask them. Can you send me your public key in tg (my handle is the same as my github)

I have posted it in the General channel and mentioned you.

[carina-akaia]

UPD: The most recent version of QueryAPI widget

[jefedeoro]

What if we use pikespeak.ai for this?
I think It possibly could help streamline the process.

[frol]

Just a note, I feel QueryAPI might be too convoluted to implement, and afterall it is still a web2 service, so we might as well run our own small search serving service or use an existing SaaS solution and use regular fetch call from BOS components.

[ailisp]

Status: I made a raw version of indexer which can index all text and do full text search. The Query API worker has some issue because of a missing block. The problem is fixed and will be released in near/queryapi#196