Skip to content

Commit

Permalink
Make it so when an embedding hasn't chagned, the metadata updated is.
Browse files Browse the repository at this point in the history
The similarCards.last_updated flow requires cards to have their point updated every time the card.updated
timestamp updates. That updates more often than the embedding changes.

This makes it so the last_updated filter should work.

Part of #646. Part of #670.
  • Loading branch information
jkomoros committed Nov 12, 2023
1 parent 847eea8 commit ccc93fe
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 5 deletions.
18 changes: 16 additions & 2 deletions functions/src/embeddings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -328,8 +328,22 @@ class EmbeddingStore {
}

if (existingPoint && existingPoint.payload && existingPoint.payload.content === text) {
//The embedding exists and is up to date, no need to do anything else.
console.log(`The embedding content had not changed for ${card.id} so stopping early`);
//The embedding exists and is up to date, no need to recompute the
//embedding. We do still want to update the timestamp in the vector
//store though, so in the future last_updated will note that it's
//current as of this moment, not when the embedding changed...
//because a card itself doesn't know the timestamp when the
//embeddable content last changed.
await this._qdrant.setPayload(QDRANT_COLLECTION_NAME, {
points: [
existingPoint.id
],
payload: {
last_updated: Date.now()
}
});

console.log(`Updated the metadata for ${card.id} because its embedding had not changed`);
return;
}

Expand Down
3 changes: 0 additions & 3 deletions src/actions/similarity.ts
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,6 @@ const similarCards = async (cardID : CardID, lastUpdated : MillisecondsSinceEpoc
};
}

//TODO: there's a massive problem with this flow currently... if a card is updated but
//doesn't have an embedding updated.

const request : SimilarCardsRequestData = {
card_id: cardID,
last_updated: lastUpdated
Expand Down

0 comments on commit ccc93fe

Please sign in to comment.