Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](memory) Fix compatibility with CgroupV2 #44579

Merged
merged 4 commits into from
Nov 27, 2024
Merged

[fix](memory) Fix compatibility with CgroupV2 #44579

merged 4 commits into from
Nov 27, 2024

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Nov 25, 2024 •
edited
Loading

What problem does this PR solve?

Problem Summary:

  1. Fix memory.max value is max.
  2. Fix memory_usage minus slab_reclaimable, part of "slab" that might be reclaimed.

Note, CgroupV2 MemAvailable is usually a little smaller than Process MemAvailable.
Process MemAvailable = MemFree - LowWaterMark + (PageCache - min(PageCache / 2, LowWaterMark)), from MemAvailable in /proc/meminfo, calculated by OS.
CgroupV2 MemAvailable = cgroup_mem_limit - cgroup_mem_usage, cgroup_mem_usage = memory.current - inactive_file - slab_reclaimable, in fact, there seems to be some memory that can be reused in cgroup_mem_usage.

ref: https://arthurchiao.art/blog/cgroupv2-zh/

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xinyiZzz
Copy link
Contributor Author

run buildall

@xinyiZzz xinyiZzz force-pushed the 20241125_fix_cgroup branch from c1b1509 to 8c25631 Compare November 25, 2024 17:21
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39760 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c25631cbdadb785a824645e054ba4e6097e3260, data reload: false

------ Round 1 ----------------------------------
q1	17719	7582	7271	7271
q2	2049	170	160	160
q3	10669	1105	1185	1105
q4	10562	710	676	676
q5	7619	2747	2689	2689
q6	240	147	145	145
q7	989	659	598	598
q8	9246	1926	1999	1926
q9	6589	6447	6448	6447
q10	7008	2295	2338	2295
q11	467	276	261	261
q12	415	210	208	208
q13	17779	3051	2981	2981
q14	246	216	214	214
q15	570	538	520	520
q16	653	597	595	595
q17	971	517	509	509
q18	7312	6771	6566	6566
q19	1351	942	982	942
q20	469	183	179	179
q21	4218	3166	3257	3166
q22	376	322	307	307
Total cold run time: 107517 ms
Total hot run time: 39760 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7256	7240	7268	7240
q2	328	230	230	230
q3	2911	2771	2969	2771
q4	2086	1764	1799	1764
q5	5676	5732	5643	5643
q6	225	141	141	141
q7	2283	1834	1810	1810
q8	3426	3604	3512	3512
q9	8850	8924	8901	8901
q10	3597	3584	3560	3560
q11	624	501	515	501
q12	849	644	614	614
q13	11469	3262	3206	3206
q14	301	268	288	268
q15	602	512	526	512
q16	721	643	654	643
q17	1878	1637	1644	1637
q18	8401	7942	7593	7593
q19	1793	1622	1651	1622
q20	2110	1877	1885	1877
q21	5665	5463	5466	5463
q22	671	603	596	596
Total cold run time: 71722 ms
Total hot run time: 60104 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.34% (9980/26032)
Line Coverage: 29.43% (83508/283780)
Region Coverage: 28.60% (42987/150303)
Branch Coverage: 25.18% (21835/86726)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8c25631cbdadb785a824645e054ba4e6097e3260_8c25631cbdadb785a824645e054ba4e6097e3260/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 197583 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c25631cbdadb785a824645e054ba4e6097e3260, data reload: false

query1	1232	922	960	922
query2	6244	2137	2056	2056
query3	10804	4111	3924	3924
query4	67818	29680	23685	23685
query5	5263	462	457	457
query6	438	183	182	182
query7	5961	286	294	286
query8	326	237	225	225
query9	9474	2741	2772	2741
query10	457	243	245	243
query11	17672	15464	15838	15464
query12	156	104	107	104
query13	1611	400	413	400
query14	11036	7315	7451	7315
query15	227	185	187	185
query16	7215	478	475	475
query17	1433	566	560	560
query18	1882	305	295	295
query19	192	153	142	142
query20	116	119	113	113
query21	210	104	104	104
query22	4633	4522	4677	4522
query23	34718	34982	34535	34535
query24	5268	2553	2574	2553
query25	496	389	388	388
query26	677	150	149	149
query27	1992	286	288	286
query28	4486	2501	2469	2469
query29	667	430	415	415
query30	223	152	149	149
query31	1039	840	821	821
query32	70	54	61	54
query33	416	334	294	294
query34	943	509	521	509
query35	874	744	721	721
query36	1095	975	959	959
query37	127	73	71	71
query38	4593	4367	4446	4367
query39	1540	1526	1475	1475
query40	206	97	107	97
query41	45	43	43	43
query42	107	101	103	101
query43	547	522	499	499
query44	1207	872	852	852
query45	188	166	170	166
query46	1164	716	724	716
query47	2030	1942	1908	1908
query48	425	327	332	327
query49	743	402	392	392
query50	848	402	398	398
query51	7478	7209	7009	7009
query52	99	88	89	88
query53	255	181	174	174
query54	517	389	401	389
query55	88	81	80	80
query56	271	269	247	247
query57	1331	1210	1205	1205
query58	246	235	233	233
query59	3185	3000	3086	3000
query60	280	264	263	263
query61	131	129	129	129
query62	801	672	682	672
query63	222	190	214	190
query64	1381	663	641	641
query65	3277	3193	3187	3187
query66	715	306	299	299
query67	15858	15705	15561	15561
query68	4198	566	546	546
query69	420	262	269	262
query70	1233	1169	1080	1080
query71	353	256	246	246
query72	6228	4025	4120	4025
query73	770	365	362	362
query74	10373	9127	9073	9073
query75	3424	2678	2674	2674
query76	1823	1047	1131	1047
query77	515	294	289	289
query78	10390	9393	9411	9393
query79	2329	593	602	593
query80	1284	443	439	439
query81	533	237	237	237
query82	727	116	121	116
query83	264	144	145	144
query84	273	71	70	70
query85	998	303	294	294
query86	403	305	300	300
query87	4788	4518	4645	4518
query88	3744	2230	2201	2201
query89	416	302	307	302
query90	2067	183	187	183
query91	131	105	104	104
query92	70	49	48	48
query93	2676	552	552	552
query94	906	302	342	302
query95	368	251	248	248
query96	629	287	279	279
query97	2862	2679	2668	2668
query98	216	191	196	191
query99	1649	1318	1316	1316
Total cold run time: 324329 ms
Total hot run time: 197583 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c25631cbdadb785a824645e054ba4e6097e3260, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.44	0.42	0.40
query6	1.14	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.54	0.51
query10	0.56	0.55	0.56
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.61	0.60	0.61
query14	2.75	2.83	2.71
query15	0.89	0.82	0.82
query16	0.39	0.39	0.38
query17	1.06	1.05	1.01
query18	0.22	0.21	0.21
query19	1.89	1.78	1.95
query20	0.01	0.02	0.02
query21	15.35	0.59	0.57
query22	2.52	2.69	1.92
query23	16.94	0.93	0.72
query24	3.04	1.57	1.55
query25	0.27	0.12	0.15
query26	0.49	0.14	0.13
query27	0.04	0.05	0.04
query28	9.95	1.10	1.07
query29	12.58	3.23	3.19
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.28	0.47	0.47
query33	3.02	3.00	3.05
query34	16.96	4.44	4.48
query35	4.62	4.49	4.48
query36	0.67	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.16 s
Total hot run time: 33.05 s

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40424 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ce27e435a2b865ae813f3ad0d38ca8eabc758c1, data reload: false

------ Round 1 ----------------------------------
q1	17657	7882	7325	7325
q2	2042	186	189	186
q3	10632	1097	1230	1097
q4	10360	789	766	766
q5	7637	2812	2800	2800
q6	240	149	151	149
q7	988	655	612	612
q8	9256	1914	2003	1914
q9	6680	6462	6427	6427
q10	7073	2329	2348	2329
q11	486	264	266	264
q12	407	216	220	216
q13	17755	3040	3039	3039
q14	268	225	225	225
q15	575	543	535	535
q16	685	579	582	579
q17	993	580	539	539
q18	7340	6790	6810	6790
q19	1372	1076	999	999
q20	495	185	180	180
q21	3965	3140	3269	3140
q22	379	313	324	313
Total cold run time: 107285 ms
Total hot run time: 40424 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7358	7261	7259	7259
q2	323	237	237	237
q3	2923	2950	2976	2950
q4	2056	1817	1884	1817
q5	5663	5696	5697	5696
q6	229	141	141	141
q7	2254	1824	1844	1824
q8	3391	3591	3563	3563
q9	8784	9030	8975	8975
q10	3591	3583	3566	3566
q11	601	511	511	511
q12	829	613	615	613
q13	10467	3291	3205	3205
q14	302	275	271	271
q15	576	528	518	518
q16	687	660	671	660
q17	1870	1679	1599	1599
q18	8347	7759	7600	7600
q19	1809	1594	1680	1594
q20	2140	1882	1895	1882
q21	5519	5388	5443	5388
q22	643	578	570	570
Total cold run time: 70362 ms
Total hot run time: 60439 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197065 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9ce27e435a2b865ae813f3ad0d38ca8eabc758c1, data reload: false

query1	1233	944	973	944
query2	6244	2050	2047	2047
query3	10815	4071	4108	4071
query4	67032	28663	23540	23540
query5	4938	467	452	452
query6	420	186	182	182
query7	5666	307	284	284
query8	321	229	231	229
query9	9344	2646	2618	2618
query10	465	243	239	239
query11	17561	15309	15811	15309
query12	151	103	102	102
query13	1563	416	414	414
query14	10310	7145	7499	7145
query15	208	177	196	177
query16	7119	501	458	458
query17	1178	566	594	566
query18	1818	325	319	319
query19	209	161	160	160
query20	120	115	124	115
query21	210	110	106	106
query22	4767	4482	4510	4482
query23	34682	34337	34414	34337
query24	5597	2548	2494	2494
query25	492	391	382	382
query26	642	157	151	151
query27	1832	281	271	271
query28	4380	2457	2471	2457
query29	666	415	415	415
query30	216	152	147	147
query31	1008	844	860	844
query32	65	56	52	52
query33	394	299	307	299
query34	929	554	522	522
query35	854	741	731	731
query36	1063	958	961	958
query37	123	70	76	70
query38	4593	4469	4524	4469
query39	1532	1470	1483	1470
query40	197	100	100	100
query41	46	42	43	42
query42	112	150	101	101
query43	526	496	490	490
query44	1205	844	852	844
query45	191	174	172	172
query46	1162	682	723	682
query47	2020	1925	1941	1925
query48	428	331	328	328
query49	717	407	405	405
query50	843	393	386	386
query51	7433	7305	7143	7143
query52	96	84	84	84
query53	252	182	171	171
query54	511	395	392	392
query55	73	73	79	73
query56	236	238	231	231
query57	1297	1175	1160	1160
query58	221	225	214	214
query59	3351	3098	2949	2949
query60	265	245	256	245
query61	107	109	107	107
query62	787	674	710	674
query63	213	187	185	185
query64	1345	668	628	628
query65	3272	3222	3204	3204
query66	700	296	295	295
query67	15957	15550	15551	15550
query68	4010	558	551	551
query69	438	249	258	249
query70	1208	1082	1144	1082
query71	355	249	253	249
query72	6434	4133	4090	4090
query73	762	357	363	357
query74	10150	8986	9072	8986
query75	3397	2694	2675	2675
query76	1860	1137	1078	1078
query77	487	261	259	259
query78	10734	9514	9524	9514
query79	2631	594	617	594
query80	1362	453	424	424
query81	544	259	238	238
query82	1278	118	127	118
query83	178	154	142	142
query84	283	74	71	71
query85	1005	309	346	309
query86	430	305	298	298
query87	4701	4824	4644	4644
query88	4115	2250	2204	2204
query89	422	303	288	288
query90	1978	193	194	193
query91	136	103	110	103
query92	70	49	53	49
query93	2907	534	530	530
query94	893	268	289	268
query95	358	247	244	244
query96	645	276	287	276
query97	2870	2678	2683	2678
query98	220	199	201	199
query99	1621	1319	1297	1297
Total cold run time: 322805 ms
Total hot run time: 197065 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.37% (9980/26007)
Line Coverage: 29.47% (83540/283491)
Region Coverage: 28.61% (42998/150289)
Branch Coverage: 25.20% (21833/86630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9ce27e435a2b865ae813f3ad0d38ca8eabc758c1_9ce27e435a2b865ae813f3ad0d38ca8eabc758c1/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 33.39 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9ce27e435a2b865ae813f3ad0d38ca8eabc758c1, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.06
query4	1.61	0.10	0.11
query5	0.44	0.41	0.41
query6	1.15	0.67	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.56	0.55	0.50
query10	0.54	0.56	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.10
query13	0.62	0.60	0.60
query14	2.86	2.71	2.74
query15	0.90	0.82	0.83
query16	0.38	0.37	0.38
query17	1.07	1.07	1.02
query18	0.22	0.21	0.21
query19	1.99	1.88	1.96
query20	0.01	0.01	0.01
query21	15.36	0.61	0.58
query22	2.69	2.12	2.53
query23	16.94	0.81	0.79
query24	2.78	1.74	1.48
query25	0.28	0.10	0.13
query26	0.57	0.14	0.13
query27	0.04	0.05	0.03
query28	9.82	1.10	1.06
query29	12.54	3.26	3.24
query30	0.24	0.06	0.06
query31	2.85	0.38	0.38
query32	3.27	0.46	0.47
query33	3.08	3.14	3.04
query34	17.11	4.49	4.49
query35	4.55	4.46	4.50
query36	0.66	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.04
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.3 s
Total hot run time: 33.39 s

@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.35% (9977/26013)
Line Coverage: 29.45% (83494/283517)
Region Coverage: 28.60% (42980/150293)
Branch Coverage: 25.21% (21835/86628)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9cc5be1cf670bff2959e6e6be1c4641a15ce650f_9cc5be1cf670bff2959e6e6be1c4641a15ce650f/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 39902 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9cc5be1cf670bff2959e6e6be1c4641a15ce650f, data reload: false

------ Round 1 ----------------------------------
q1	17621	7504	7326	7326
q2	2050	176	177	176
q3	10685	1075	1183	1075
q4	10532	728	673	673
q5	7590	2746	2753	2746
q6	241	151	151	151
q7	989	628	607	607
q8	9262	1882	1947	1882
q9	6592	6371	6478	6371
q10	6976	2285	2305	2285
q11	473	263	271	263
q12	406	221	214	214
q13	17771	2999	3086	2999
q14	245	212	208	208
q15	572	535	515	515
q16	678	590	588	588
q17	984	478	473	473
q18	7326	6707	6736	6707
q19	1347	1009	999	999
q20	481	181	188	181
q21	3994	3207	3151	3151
q22	387	330	312	312
Total cold run time: 107202 ms
Total hot run time: 39902 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7256	7228	7219	7219
q2	330	231	231	231
q3	2893	2842	2948	2842
q4	2057	1816	1752	1752
q5	5644	5654	5665	5654
q6	227	139	139	139
q7	2249	1815	1815	1815
q8	3375	3647	3465	3465
q9	8980	8825	8898	8825
q10	3581	3548	3554	3548
q11	592	518	503	503
q12	863	604	644	604
q13	10289	3312	3289	3289
q14	304	280	275	275
q15	575	531	534	531
q16	673	644	645	644
q17	1849	1642	1616	1616
q18	8422	7754	7690	7690
q19	1679	1547	1525	1525
q20	2131	1889	1886	1886
q21	5502	5402	5387	5387
q22	616	595	567	567
Total cold run time: 70087 ms
Total hot run time: 60007 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195542 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9cc5be1cf670bff2959e6e6be1c4641a15ce650f, data reload: false

query1	1293	961	971	961
query2	6277	2119	1998	1998
query3	10802	4025	3806	3806
query4	66952	28765	23596	23596
query5	5041	448	457	448
query6	412	183	175	175
query7	5541	301	289	289
query8	318	231	223	223
query9	8819	2671	2639	2639
query10	444	258	264	258
query11	17183	15262	15856	15262
query12	157	108	107	107
query13	1479	442	420	420
query14	10528	6951	7712	6951
query15	208	186	197	186
query16	7092	431	486	431
query17	1063	576	560	560
query18	1835	294	307	294
query19	194	150	157	150
query20	123	118	122	118
query21	205	104	107	104
query22	4672	4414	4498	4414
query23	34779	34040	34464	34040
query24	5415	2497	2492	2492
query25	495	392	394	392
query26	660	151	150	150
query27	1978	281	285	281
query28	4123	2487	2454	2454
query29	697	443	427	427
query30	214	150	163	150
query31	1024	840	845	840
query32	69	55	59	55
query33	485	298	289	289
query34	923	502	505	502
query35	885	755	731	731
query36	1087	964	980	964
query37	123	76	74	74
query38	4430	4578	4364	4364
query39	1518	1486	1501	1486
query40	218	104	111	104
query41	51	48	44	44
query42	108	102	101	101
query43	550	496	524	496
query44	1203	839	827	827
query45	191	167	174	167
query46	1163	730	696	696
query47	2031	1956	1910	1910
query48	416	305	329	305
query49	735	381	401	381
query50	851	395	385	385
query51	7286	7121	7108	7108
query52	98	88	89	88
query53	255	175	174	174
query54	524	395	396	395
query55	79	77	76	76
query56	253	226	283	226
query57	1299	1180	1131	1131
query58	213	217	204	204
query59	3223	3190	2821	2821
query60	269	274	237	237
query61	149	109	108	108
query62	782	664	647	647
query63	222	184	184	184
query64	1356	657	626	626
query65	3285	3223	3252	3223
query66	699	302	310	302
query67	16122	15679	15760	15679
query68	4142	567	551	551
query69	435	249	252	249
query70	1163	1122	1087	1087
query71	361	244	243	243
query72	6402	4069	4128	4069
query73	756	349	352	349
query74	10380	8966	9023	8966
query75	3415	2664	2623	2623
query76	1914	1117	996	996
query77	494	270	269	269
query78	10550	9458	9385	9385
query79	3005	607	602	602
query80	1408	419	444	419
query81	533	234	230	230
query82	1023	115	121	115
query83	260	150	146	146
query84	290	84	71	71
query85	1199	299	299	299
query86	489	311	304	304
query87	4773	4611	4556	4556
query88	4455	2145	2129	2129
query89	420	284	303	284
query90	1999	184	185	184
query91	133	137	104	104
query92	73	53	53	53
query93	2784	548	553	548
query94	888	308	293	293
query95	356	248	249	248
query96	640	283	273	273
query97	2886	2684	2712	2684
query98	218	194	202	194
query99	1564	1313	1293	1293
Total cold run time: 322582 ms
Total hot run time: 195542 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9cc5be1cf670bff2959e6e6be1c4641a15ce650f, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.11	0.11
query5	0.42	0.41	0.44
query6	1.19	0.66	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.51
query10	0.54	0.56	0.56
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.60	0.61	0.59
query14	2.81	2.85	2.78
query15	0.90	0.84	0.82
query16	0.38	0.37	0.38
query17	1.02	0.98	1.05
query18	0.22	0.21	0.20
query19	1.86	1.90	1.94
query20	0.01	0.01	0.01
query21	15.37	0.59	0.60
query22	2.75	2.18	1.46
query23	16.97	0.81	0.87
query24	3.03	0.29	1.62
query25	0.17	0.18	0.16
query26	0.43	0.14	0.14
query27	0.03	0.05	0.04
query28	11.01	1.09	1.07
query29	12.57	3.19	3.23
query30	0.24	0.06	0.06
query31	2.86	0.37	0.37
query32	3.29	0.46	0.47
query33	2.98	2.99	2.98
query34	17.04	4.44	4.44
query35	4.52	4.52	4.45
query36	0.67	0.49	0.51
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.12	0.12
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.31 s
Total hot run time: 31.57 s

wangbo
wangbo approved these changes Nov 27, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit 23df40b into apache:master Nov 27, 2024
21 of 23 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 3, 2024
@xinyiZzz
[fix](memory) Fix compatibility with CgroupV2 (#44579)
2b5418a 
### What problem does this PR solve?

Problem Summary:

1. Fix `memory.max` value is `max`.
2. Fix `memory_usage` minus `slab_reclaimable`, part of "slab" that
might be reclaimed.

Note, CgroupV2 MemAvailable is usually a little smaller than Process
MemAvailable.
Process `MemAvailable = MemFree - LowWaterMark + (PageCache -
min(PageCache / 2, LowWaterMark))`, from `MemAvailable` in
`/proc/meminfo`, calculated by OS.
CgroupV2 `MemAvailable = cgroup_mem_limit - cgroup_mem_usage`,
`cgroup_mem_usage = memory.current - inactive_file - slab_reclaimable`,
in fact, there seems to be some memory that can be reused in
`cgroup_mem_usage`.

ref: https://arthurchiao.art/blog/cgroupv2-zh/
@github-actions github-actions bot mentioned this pull request Dec 3, 2024
Merged
github-actions bot pushed a commit that referenced this pull request Dec 3, 2024
@xinyiZzz
[fix](memory) Fix compatibility with CgroupV2 (#44579)
cc8ae3b 
### What problem does this PR solve?

Problem Summary:

1. Fix `memory.max` value is `max`.
2. Fix `memory_usage` minus `slab_reclaimable`, part of "slab" that
might be reclaimed.

Note, CgroupV2 MemAvailable is usually a little smaller than Process
MemAvailable.
Process `MemAvailable = MemFree - LowWaterMark + (PageCache -
min(PageCache / 2, LowWaterMark))`, from `MemAvailable` in
`/proc/meminfo`, calculated by OS.
CgroupV2 `MemAvailable = cgroup_mem_limit - cgroup_mem_usage`,
`cgroup_mem_usage = memory.current - inactive_file - slab_reclaimable`,
in fact, there seems to be some memory that can be reused in
`cgroup_mem_usage`.

ref: https://arthurchiao.art/blog/cgroupv2-zh/
@github-actions github-actions bot mentioned this pull request Dec 3, 2024
Merged
dataroaring pushed a commit that referenced this pull request Dec 4, 2024
@github-actions @xinyiZzz
branch-3.0: [fix](memory) Fix compatibility with CgroupV2 #44579 (#44933
9a00544 
)

Cherry-picked from #44579

Co-authored-by: Xinyi Zou <[email protected]>
yiguolei pushed a commit that referenced this pull request Dec 4, 2024
@github-actions @xinyiZzz
branch-2.1: [fix](memory) Fix compatibility with CgroupV2 #44579 (#44934
c4bd0e8 
)

Cherry-picked from #44579

Co-authored-by: Xinyi Zou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wangbo wangbo wangbo approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@xinyiZzz @doris-robot @yiguolei @wangbo @dataroaring