Implement server side search

[ailisp]

Implement #115. Fix #292 #248 #106

The search code is in QueryAPI, code is public here: https://near.org/dataplatform.near/widget/QueryApi.App?selectedIndexerPath=bo.near/devhub_v17&view=editor-window can be forked and updated if there's update in post add / edit format.

Preview

https://near.social/bo.near/widget/gigs-board.pages.Feed?nearDevGovGigsContractAccountId=devgovgigs.near

Testing

search one of author, tag or content; search author + tag, author + content, tag + content; search author + tag + content. All testing produce correct result.

Notes

• A design for search result pagination is needed to have that pagination on the UI. It currently hard coded to display top 100 search result, to make server response faster enough. In practice, it's not a problem because user is very unlikely to browse more than 100 search result instead of trying to refine the search. So I think pagination isn't a priority. Pagination on server is implemented.
• server has more capability than frontend shows, for example filter by post category, sort with some criteria, search top level posts only or search posts & comments (current behavior), title, history versions, sponsor, editors, etc. These can be easily enabled if design. We may need to think of a new design of the search panel.
• A "Search" button is added to trigger search to avoid the previous stuck when typing experience. Also due to a limitation of VM we cannot trigger search with <Enter> key. To make the search experience consistent, now search by author or tag also doesn't trigger until you click "Search" button
• Same as the behavior before, you also must click the auto-completion item, to search by author or tag.

doesn't set author to `frol.near` does set author and now click search can search author by `frol.near`

This is suboptimal and hopefully we can find some workaround to change this <Typeahead> behavior

[frol]

@ailisp Does the search work for the post titles? I cannot find the fellowship program announcement post with either of the following queries: "devdao fellowship", "fellowship program"

[ailisp]

@frol it does not. it's support on server but the UI is search by content only.

Search by title need some design, because for a title: "Sponsorship: XXX" the title is actually only "XXX" and it could be confused to not see if they search for "Sponsorship" and didn't get it. It's also support on server side to search by post category, e.g. "Sponsorship". How to present these search criteria and make them all intuitive, no confusion requires more think about the search panel. So I keep search panel minimally changed for now.

[petersalomonsen]

How do we ensure that the query is updated if someone changes the post format. Should we add a test for the query? Also I find the reason for #292 to be that some posts does not deserialize anymore. This will still be a problem, and not fixed by this PR. Maybe we should rephrase #292 since you are now changing to another way of searching?

[petersalomonsen]

I think this looks very good! If possible to add a playwright test ( checking that the query works ), then that would be great. But I've tested the preview and I think it works fine.

Let me know what to do with #292 . The bug in the contract is still there, and I can still look into it if we want to.

[ailisp]

How do we ensure that the query is updated if someone changes the post format.

So there isn't a good way to ensure that. This approaches for all transactions does a add_post or edit_post function call. We will need to manually read all contract changes to the parameter type of these two methods.

Regarding tests, yes would be nice to add one. I'll make sure to some either on this one or a later PR where we'll add more criteria to search. For #292, I looked at your comments on #292. It's a great investigation, however it seems like not the bug that @ori-near reported. Ori said search any term fail to find result, not just post 146. Also search for post 146 works on this approach:

The reason is it looks for transaction histories of successful add_post and edit_post. So even certain corruption in state wouldn't affect search the post.

I think we should continue investigate the corruption of state issue, starting from reproduce of devgovgigs.near.get_post({"post_id": 146}). Let's create a new issue to track it, and do ensure not just post 146, but using a script to identify all problematic cause and fix them. Ideally also identify when and why they're corrupted and prevent it to happen again in future

[petersalomonsen]

It's a great investigation, however it seems like not the bug that @ori-near reported. Ori said search any term fail to find result, not just post 146.

The current search functionality use get_posts which currently does not work because of some posts failing to deserialize ( 146 among others ). So that's why it does not find any results, but anyway that problem of getting results will be solved by your PR here.

[ailisp]

I added one search test with @petersalomonsen's testing framework and will merge this. Will talk to @ori-near on more search criteria: filter by post category, sort with some criteria, search top level posts only or search posts & comments (current behavior), title, history versions, sponsor, editors, etc. They are easy to add but we need to figure out what we need.