Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to configure partition range of event tables #14474

Open
4 of 9 tasks
AlanCoding opened this issue Sep 21, 2023 · 1 comment
Open
4 of 9 tasks

Ability to configure partition range of event tables #14474

AlanCoding opened this issue Sep 21, 2023 · 1 comment

Comments

@AlanCoding
Copy link
Member

Please confirm the following

  • I agree to follow this project's code of conduct.
  • I have checked the current issues for duplicates.
  • I understand that AWX is open source software provided for free and that I might not receive a timely response.

Feature type

New Feature

Feature Summary

We partition event (job/project/etc.) tables hourly. This hourly time frame is hard-coded.

awx/awx/main/utils/common.py

Lines 1156 to 1157 in 6d3c22e

start = start.replace(microsecond=0, second=0, minute=0)
end = start + timedelta(hours=1)

This issue asks that we allow this to be configurable, probably as an ordinary file-based setting. Doing this requires two main things be accomplished:

  1. Basic code changes so that we don't use a hard-coded 1-hour partition range
  2. Migration steps that users need to follow so that they can enact this change without bad things happening

Some research is needed to determine what hazards we may hit mixing partition ranges. There's also concern about overlapping partitions for the currently active partition when the setting is changed. If a user changes from hourly to daily partitions, then the next job could create a partition for the day that overlaps with already-created partitions for the first few hours of that day.

Select the relevant components

  • UI
  • API
  • Docs
  • Collection
  • CLI
  • Other

Steps to reproduce

Try to configure the partition range

Current results

Code changes required

Sugested feature result

new setting

Additional information

No response

@AlanCoding
Copy link
Member Author

Reading here...

https://stackoverflow.com/questions/31553966/is-it-safe-to-use-postgres-table-partitioning-with-overlapping-constraints

Now I'm wondering if, instead of migrating data, we can let the prior partition size play out. Consider 2 scenarios:

  1. Going from hourly to daily partitions
  2. Going from daily to hourly partitions

Imagine that the time is 1:01am, so if you start on hourly partitions (case (1)), you have 1 partition created and would be creating the 2nd for that day. Instead, since the setting was changed (presumably at 12:59am), you try to create a daily partition and hit the error. The problem here is that you would have to save the prior partition value somewhere so that you could reference it until a day has passed. If you have this, you could catch the error, and then try creating the partition using the old range. This should work for case (1).

In case (2), you would prefer to ignore the error. In the hypothetical, a partition would already have been created before 12:59am for the entire day. It should work to create no new partitions until the next day starts. Perhaps we could just catch the "overlapping partitions" error twice, and do nothing in that case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants