Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

POC: Feat/make indexing more resiliant #16546

Open
wants to merge 2 commits into
base: v13/dev
Choose a base branch
from

Conversation

bielu
Copy link
Contributor

@bielu bielu commented Jun 3, 2024

Prerequisites

  • I have added steps to test this contribution in the description below

If there's an existing issue for this PR then this fixes

Description

This Pr is POC for solving issue with rebuilding of indexes on startup. Also adding small flexibility around page size when reindexing as current we pulled 10k nodes, which if they have more than 100 properties might cause really slow indexing when we pull more than 1k. I am making it as POC as won't spend more time on this pr unless HQ confirm this is the way and I should improve code in this pr.

Copy link

github-actions bot commented Jun 3, 2024

Hi there @bielu, thank you for this contribution! 👍

While we wait for one of the Core Collaborators team to have a look at your work, we wanted to let you know about that we have a checklist for some of the things we will consider during review:

  • It's clear what problem this is solving, there's a connected issue or a description of what the changes do and how to test them
  • The automated tests all pass (see "Checks" tab on this PR)
  • The level of security for this contribution is the same or improved
  • The level of performance for this contribution is the same or improved
  • Avoids creating breaking changes; note that behavioral changes might also be perceived as breaking
  • If this is a new feature, Umbraco HQ provided guidance on the implementation beforehand
  • 💡 The contribution looks original and the contributor is presumably allowed to share it

Don't worry if you got something wrong. We like to think of a pull request as the start of a conversation, we're happy to provide guidance on improving your contribution.

If you realize that you might want to make some changes then you can do that by adding new commits to the branch you created for this work and pushing new commits. They should then automatically show up as updates to this pull request.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

@nul800sebastiaan
Copy link
Member

Thanks @bielu - I have noted this PR and asked for the team to have a look and see if it's the right direction. Of course we have a "little" conference coming up next week so it's taking a bit longer to get to.

/// <summary>
///
/// </summary>
public class IndexRebuildStatusManager : IIndexRebuildStatusManager
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My recommended approach would be to do this: Shazwazza/Examine#372 (comment)

The only real way to know if the indexing is done in a resilient way would be to have an actual document in the index certifying that rebuilding is successful instead of relying on in-memory cache which is problematic.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Shazwazza I am not 100% convinced about usage of additional index, as we both know less indexes is actually better with lucene. I am thinking maybe we should use additonal sql table, as it will be eqally resiliant as using index, but it will not require us to create index, what you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bielu Sorry, I probably wasn't clear in my suggestion. We don't want to use an extra index to store any data, we can just use a marker document within the index. For example:

  • Rebuilding an index deletes all data
  • The index is populated with the normal data
  • When the IndexPopulator is done populating the index, it then writes a special marker document signaling that the populator is done. Perhaps this document has a field like __Populated: y

Then the rebuild checker, just checks if the document count for __Populated: y == 1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Shazwazza that's make sense now! We can also extend it to check what populator are registered to show how many of them is done! I will make update to this pr

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Shazwazza i started changing implementation of this service to use examine underhood, can you have quick look and check if that is what you had in mind?
as in this way now we can also repeat failed batches (but i think I will need play around little more)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants