Add reshuffle kmeans actor scan #9725

MBkkt · 2024-09-24T16:27:14Z

No description provided.

github-actions · 2024-09-24T16:29:54Z

⚪ 2024-09-24 16:29:53 UTC Pre-commit check linux-x86_64-release-clang14 for 37368b3 has started.
⚪ 2024-09-24 16:30:22 UTC Artifacts will be uploaded here
⚪ 2024-09-24 16:33:47 UTC ya make is running...
🟢 2024-09-24 17:24:52 UTC Build successful.

github-actions · 2024-09-24T16:30:20Z

⚪ 2024-09-24 16:30:20 UTC Pre-commit check linux-x86_64-release-asan for 37368b3 has started.
⚪ 2024-09-24 16:30:30 UTC Artifacts will be uploaded here
⚪ 2024-09-24 16:33:49 UTC ya make is running...
🔴 2024-09-24 18:53:55 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14562	14352	0	35	52	123

🟢 2024-09-24 18:55:11 UTC Build successful.
🟡 2024-09-24 18:55:44 UTC ydbd size 5.7 GiB changed* by +2.0 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `1a5bc4d`	merge: `37368b3`	diff	diff %
ydbd size	6 080 405 296 Bytes	6 082 472 984 Bytes	+2.0 MiB	+0.034%
ydbd stripped size	1 520 074 256 Bytes	1 520 528 176 Bytes	+443.3 KiB	+0.030%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-24T16:30:25Z

⚪ 2024-09-24 16:30:25 UTC Pre-commit check linux-x86_64-relwithdebinfo for 37368b3 has started.
⚪ 2024-09-24 16:30:36 UTC Artifacts will be uploaded here
⚪ 2024-09-24 16:33:54 UTC ya make is running...
🟡 2024-09-24 18:25:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
77131	63028	0	7	14061	35

⚪ 2024-09-24 18:32:36 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-24 18:44:06 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
287 (only retried tests)	189	0	0	8	90

🟢 2024-09-24 18:44:14 UTC Build successful.
🟡 2024-09-24 18:44:55 UTC ydbd size 8.5 GiB changed* by +1.9 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `2b5f88d`	merge: `37368b3`	diff	diff %
ydbd size	9 073 512 024 Bytes	9 075 543 848 Bytes	+1.9 MiB	+0.022%
ydbd stripped size	490 601 672 Bytes	490 725 576 Bytes	+121.0 KiB	+0.025%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-25T12:44:44Z

⚪ 2024-09-25 12:44:44 UTC Pre-commit check linux-x86_64-release-asan for e4126db has started.
⚪ 2024-09-25 12:45:03 UTC Artifacts will be uploaded here
⚪ 2024-09-25 12:48:31 UTC ya make is running...
⚫ 2024-09-25 14:12:13 UTC Check cancelled

github-actions · 2024-09-25T12:44:45Z

⚪ 2024-09-25 12:44:44 UTC Pre-commit check linux-x86_64-release-clang14 for e4126db has started.
⚪ 2024-09-25 12:45:04 UTC Artifacts will be uploaded here
⚪ 2024-09-25 12:48:34 UTC ya make is running...
🟢 2024-09-25 13:36:07 UTC Build successful.

github-actions · 2024-09-25T12:44:50Z

⚪ 2024-09-25 12:44:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for e4126db has started.
⚪ 2024-09-25 12:45:00 UTC Artifacts will be uploaded here
⚪ 2024-09-25 12:48:18 UTC ya make is running...
⚫ 2024-09-25 14:12:26 UTC Check cancelled

snaury · 2024-09-25T12:38:40Z

ydb/core/tx/datashard/kmeans_helper.h

+
+namespace NKikimr::NDataShard::NKMeans {
+
+// TODO(mbkkt) We are know all tempalte parameters for these functions and classes so we can move them to cpp


Много опечаток, и совершенно не ясно, что здесь имеется ввиду. Если эти функции используются только в одном .cpp, то можно унести в .cpp. Если не только в одном .cpp, то не ясно что ты хочешь здесь делать и зачем. Вроде нет ничего плохого, что эти функции в .h, ты ведь вряд ли хочешь для нескольких TRes писать несколько реализаций. Мне кажется этот комментарий лишний.

Тут скорее про TCalculation.
Типо в принципе если функцию, которая делает инстанциацию шаблона (свитч по вектор сеттингс) убрать в kmeans_helper.cpp, то эти функции тоже можно унести туда.
Сейчас оно по сути два раза инстациируется (local_kmeans и reshuffle_kmeans)

Просто я пока не придумал красивого способа это сделать (без явного перечисления всех комбинаций)

Но удалил коммент.

ydb/core/tx/datashard/reshuffle_kmeans.cpp

ydb/core/tx/datashard/datashard_ut_reshuffle_kmeans.cpp

github-actions · 2024-09-25T14:15:05Z

⚪ 2024-09-25 14:15:05 UTC Pre-commit check linux-x86_64-release-clang14 for ae83add has started.
⚪ 2024-09-25 14:15:14 UTC Artifacts will be uploaded here
⚪ 2024-09-25 14:18:41 UTC ya make is running...
🟢 2024-09-25 15:07:22 UTC Build successful.

github-actions · 2024-09-25T14:16:31Z

⚪ 2024-09-25 14:16:30 UTC Pre-commit check linux-x86_64-release-asan for ae83add has started.
⚪ 2024-09-25 14:16:40 UTC Artifacts will be uploaded here
⚪ 2024-09-25 14:20:07 UTC ya make is running...
🟡 2024-09-25 16:44:17 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14569	14356	0	38	56	119

🟢 2024-09-25 16:45:33 UTC Build successful.
🟡 2024-09-25 16:46:10 UTC ydbd size 5.7 GiB changed* by +983.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `d59377c`	merge: `ae83add`	diff	diff %
ydbd size	6 081 793 032 Bytes	6 082 800 560 Bytes	+983.9 KiB	+0.017%
ydbd stripped size	1 520 304 592 Bytes	1 520 540 208 Bytes	+230.1 KiB	+0.015%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-25T14:16:59Z

⚪ 2024-09-25 14:16:58 UTC Pre-commit check linux-x86_64-relwithdebinfo for ae83add has started.
⚪ 2024-09-25 14:17:09 UTC Artifacts will be uploaded here
⚪ 2024-09-25 14:20:36 UTC ya make is running...
🟡 2024-09-25 16:15:26 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
77126	63151	0	13	13844	118

⚪ 2024-09-25 16:22:40 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-25 16:34:07 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
149 (only retried tests)	56	0	1	0	92

⚪ 2024-09-25 16:34:16 UTC ya make is running... (failed tests rerun, try 3)
🟢 2024-09-25 16:45:33 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
98 (only retried tests)	8	0	0	0	90

🟢 2024-09-25 16:45:42 UTC Build successful.
🟡 2024-09-25 16:46:20 UTC ydbd size 8.5 GiB changed* by +1.5 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `d59377c`	merge: `ae83add`	diff	diff %
ydbd size	9 076 657 792 Bytes	9 078 261 376 Bytes	+1.5 MiB	+0.018%
ydbd stripped size	490 716 488 Bytes	490 781 576 Bytes	+63.6 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

WIP

9dca808

MBkkt added 2 commits September 25, 2024 11:48

Code dedup

ba5240a

Move some functions to separate file

da53aee

This comment was marked as outdated.

Sign in to view

Insert final newline

44f8ee3

snaury reviewed Sep 25, 2024

View reviewed changes

MBkkt added 3 commits September 25, 2024 13:32

Review suggestion

fde5d70

Format columnlimit 120

e86e36d

Manual format multiline function declaration

2d2c633

MBkkt requested a review from snaury September 26, 2024 08:48

snaury approved these changes Sep 26, 2024

View reviewed changes

MBkkt merged commit b9e1b38 into ydb-platform:main Sep 26, 2024
12 checks passed

MBkkt self-assigned this Sep 27, 2024

This was referenced Sep 29, 2024

YQ-3704 RD use CH udf in json parsing #9878

Merged

YQ-3703 pushdown for extract members in PQ provider #9939

Merged

GrigoriyPA mentioned this pull request Oct 8, 2024

YQ-3722 add simdjson parser into RD #10204

Merged

MBkkt mentioned this pull request Oct 17, 2024

👑 Global vector index #8967

Open

13 tasks

This was referenced Oct 21, 2024

Fix kafka cdc read and login without @ #10678

Merged

Allow short auto partitioning strategies yql #10701

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add reshuffle kmeans actor scan #9725

Add reshuffle kmeans actor scan #9725

MBkkt commented Sep 24, 2024

github-actions bot commented Sep 24, 2024 •

edited

Loading

github-actions bot commented Sep 24, 2024 •

edited

Loading

github-actions bot commented Sep 24, 2024 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading

snaury Sep 25, 2024

MBkkt Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading


		namespace NKikimr::NDataShard::NKMeans {

		// TODO(mbkkt) We are know all tempalte parameters for these functions and classes so we can move them to cpp

Add reshuffle kmeans actor scan #9725

Add reshuffle kmeans actor scan #9725

Conversation

MBkkt commented Sep 24, 2024

github-actions bot commented Sep 24, 2024 • edited Loading

github-actions bot commented Sep 24, 2024 • edited Loading

github-actions bot commented Sep 24, 2024 • edited Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Sep 25, 2024 • edited Loading

github-actions bot commented Sep 25, 2024 • edited Loading

github-actions bot commented Sep 25, 2024 • edited Loading

snaury Sep 25, 2024

Choose a reason for hiding this comment

MBkkt Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Sep 25, 2024 • edited Loading

github-actions bot commented Sep 25, 2024 • edited Loading

github-actions bot commented Sep 25, 2024 • edited Loading

github-actions bot commented Sep 24, 2024 •

edited

Loading

github-actions bot commented Sep 24, 2024 •

edited

Loading

github-actions bot commented Sep 24, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading

MBkkt Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024 •

edited

Loading