[Keyword search] Webcrawler folders update & backfill #9515
Conversation
| async (folder) => { | ||
| logger.info({ | ||
| folder, | ||
| execute, |
There was a problem hiding this comment.
nit: I'd log the result of getParents here to make sure nothing's going seriously wrong
| if (createdFolders.has(folder)) { | ||
| continue; | ||
| } | ||
|
|
||
| const logicalParent = isTopFolder(request.url) | ||
| ? null | ||
| : getFolderForUrl(folder); | ||
| await WebCrawlerFolder.upsert({ | ||
| const [webCrawlerFolder] = await WebCrawlerFolder.upsert({ |
There was a problem hiding this comment.
nit: in the internalId passed here I don't love the fact that we call stableIdForUrl again since we could get it from getParentsForPage (in the original code)
There was a problem hiding this comment.
Fair point, but the alternative parentFolderIds[index+1] is not very satisfying either, not very readable, etc. Maybe marginally better but not very clear
Also, outside the scope of the PR so will leave as is 👍
There was a problem hiding this comment.
If you extracted the parents in a variable and do parents[0] that would make of both worlds no? (Now possible because we do the slice)
philipperolet
force-pushed
the
webcrawler-folders
branch
from
05859b5 to
80ca9cd
Compare
Description
Fixes https://github.com/dust-tt/tasks/issues/1790
Risk
na, messing up folders but they're unused
Deploy Plan
deploy connectors
run backfill