Improve search

[dfberry]

When I search for ORM, the first page (any type of sort) doesn't show diesel. As a new Rust developer, I trust crates.io to help me find the industry leading crates. Can the search mechanism be updated to help with this?

[Turbo87]

Can the search mechanism be updated to help with this?

yeah, but it's a non-trivial amount of work and so far nobody had time to figure out how exactly to improve our search experience 😅

[eth3lbert]

Apologies for the delayed response. I just returned from vacation.

Typically, we can categorize reported search issues into two categories: 1. missing crate, 2. ordering. This discussion focuses on the ordering issue.

Currently, we default to relevance ordering, which order by rank (descending) followed by name (ascending). Therefore, if multiple crates have the same rank, the final order is determined by the lexical order of the crate's name. However, I believe most users would prefer more popular crates (as yet undefined) to appear first.

I'm uncertain whether we should modify the existing relevance ordering or create a new ordering that also considers popularity.

---

Regarding rank, we utilize the PostgreSQL ts_rank_cd function. Our searchable index is currently composed of a weighted tsvector of the following:

weight/data source:
A: name
B: keywords
C: description
D: readme

[gaocegege]

Maybe you could have a try https://github.com/tensorchord/pgvecto.rs, a vector search extension written by Rust.

[Turbo87]

@gaocegege unfortunately our database hosting provider does not allow using custom Pg extensions. same for most (all?) other hosting providers we've looked at lately.

[gaocegege]

https://pgvecto.rs/ We provide the hosted service for it, just FYI. Besides this, supabase and AWS Aurora support pgvector.

[dfberry]

I propose a new ordering for community relevance weighted as:

A: all-time downloads, recent downloads, and last updated (combined value)
B: name
C: keywords
D: description
E: readme

By adding this as a new feature, it shouldn't impact any existing backend or frontend functionality.

[TheGP]

A. Maybe no need for all-time downloads at all? At least it should be not the first priority.

An example is query "dotenv", crates.io suggest outdated and useless dotenv but there is "dotenvy" package, which is maintained fork. The website https://lib.rs/search?q=dotenv gives the correct output, there is no "dotenv" library in the results, and "dotenvy" is on the first place.

Anyway, in the meantime I suggest everyone switch to https://lib.rs/

[TheGP]

As they have been talking about search improvements for 10 years already, in the meantime I suggest everyone switch to https://lib.rs/ crates.io search is useless.

[eth3lbert]

Before we change how records are sorted, I was thinking we could consider introducing an "unmaintained" label (similar to lib.rs)? This would be especially useful for implicit unmaintained packages that are reported in the advisory database (e.g. dotenv).