feat: allow fuzzy-search on credential identifiers #3526
Conversation
identity/handler.go
Outdated
| // | ||
| // required: false | ||
| // in: query | ||
| CredentialsIdentifierSimilar string `json:"credentials_identifier_similar"` |
There was a problem hiding this comment.
WDYT of prefixing this as preview or something? We will probably need a separate search endpoint to deal with different use cases like:
- identifier search
- verifiable /recovery addresses search
- traits search
Alternatively this could also be added to a new endpoint marked as experimental. For me it's primarily about finding the way we will be solving search in Ory to get a consistent API experience. That's why I think that this should be somehow marked as: this api will be changed once we have an appropriate search design doc
| // Follow up: add page token support here, will be easy. | ||
| paginator := pop.NewPaginator(params.Page, params.PerPage) | ||
|
|
||
| if err := con.RawQuery(fmt.Sprintf(`SELECT DISTINCT identities.* |
There was a problem hiding this comment.
This query is now build here because pop does not allow DISTINCT queries. We need this, because in many cases the query would return an identity twice, once for the webauthn and once for the password credential type.
| @@ -0,0 +1 @@ | |||
| CREATE INDEX IF NOT EXISTS identity_credential_identifiers_nid_identifier_gin ON identity_credential_identifiers USING GIN (nid, identifier gin_trgm_ops); | |||
There was a problem hiding this comment.
To use online schema changes with cockroach, we recommend to manually copy this index definition and run it before applying migrations.
| default: | ||
| identifier = "%" + identifier + "%" | ||
| identifierOperator = "LIKE" |
There was a problem hiding this comment.
This is the mysql and sqlite case. Mysql does support full-text and fuzzy search, and special index types to allow them. However, it was not easy to understand at all, and I found no easy way to have the same behavior as postgres/CRDB. I think this is already an improvement on the current situation, and this feature is open for contributions!
test/schema/fixtures/config.schema.test.success/selfServiceAfterSettings.full.yaml
Outdated
Show resolved
Hide resolved
test/schema/fixtures/config.schema.test.success/selfServiceAfterSettingsAuthMethod.full.yaml
Outdated
Show resolved
Hide resolved
test/schema/fixtures/config.schema.test.success/selfServiceAfterSettingsMethod.full.yaml
Outdated
Show resolved
Hide resolved
| @@ -28,7 +28,7 @@ jobs: | |||
| - sdk-generate | |||
| services: | |||
| postgres: | |||
| image: postgres:9.6 | |||
| image: postgres:11.8 | |||
There was a problem hiding this comment.
Postgres 9 is EoL by now, and 11 is currently the oldest supported version.
Codecov Report
@@ Coverage Diff @@
## master #3526 +/- ##
=======================================
Coverage 78.68% 78.68%
=======================================
Files 342 342
Lines 23470 23470
=======================================
Hits 18467 18467
Misses 3639 3639
Partials 1364 1364 |
zepatrik
force-pushed
the
feat/identity-search-similarity
branch
from
3319f35
to
5483892
Compare
This PR adds the ability to search for sub-strings and similar strings in credential identifiers.
Note that the postgres and CRDB migrations create special indexes useful for this feature. To use online schema changes with cockroach, we recommend to manually copy the index definition and run it before applying migrations. The migration will then be a no-op.
If you run on mysql (or sqlite), no special index is created. If desired, you can create such an index manually, and it would be highly appreciated if you could contribute its definition 🙏
This feature is a preview and might change in behavior! Similarity search is not expected to return deterministic results anyway, but are only really useful for humans.