Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use jemalloc by default #2222

Closed
wants to merge 24 commits into from
Closed

Use jemalloc by default #2222

wants to merge 24 commits into from

Conversation

paulhauner
Copy link
Member

@paulhauner paulhauner commented Feb 24, 2021 •
edited by michaelsproul
Loading

Issue Addressed

NA

Proposed Changes

Builds upon @realbigsean's work in #2195 and adds:

  • jemalloc by default, use the sysalloc feature to turn it off (I wanted all the tests to run with jemalloc by default).
  • Configuration is inside LH now (e.g., narenas:1).

Additional Info

Highly experimental, do not merge. Do not use in production.

realbigsean and others added 24 commits February 10, 2021 18:14
@realbigsean
test add jemalloc
db6fda6
@realbigsean
test add jemalloc metrics
3c7920b
@realbigsean
add jemalloc deps
1d1fc6a
@realbigsean
test add jemalloc metrics
abf5cac
@realbigsean
test add jemalloc metrics
632bff2
@realbigsean
add more jemalloc stats
034b677
@realbigsean
import arenas
6f17971
@realbigsean
fix arenas
8805ef9
@realbigsean
enable background threads
074ce1f
@realbigsean
update to associated function
3408ee1
@realbigsean
semicolon
9e7a6b7
@realbigsean
fix error handling and remove background threads
ae8b61a
@realbigsean
delete background threads
fd17192
@realbigsean
delete unused dep
f205c18
@realbigsean
cargo lock
a0d7119
@realbigsean
move jemalloc to a feature
ce1c4e1
@realbigsean
add newlines
ff14ee0
@realbigsean
Merge branch 'unstable' of https://github.com/sigp/lighthouse into je…
5e968df 
…malloc

� Conflicts:
�	Cargo.lock
�	beacon_node/http_metrics/Cargo.toml
�	beacon_node/network/src/attestation_service/tests/mod.rs
@paulhauner
Make uses jemmaloc
d4010e3
@paulhauner
Merge branch 'unstable' into jemalloc-sea
4331ef8
@paulhauner
Set jemalloc as default, configure arenas
42eb539
@paulhauner
Merge branch 'unstable' into jemalloc-sea
6680b6d
@paulhauner
Fix config
88e54ab
@paulhauner
Jemalloc by default
345802f
@paulhauner paulhauner mentioned this pull request Feb 24, 2021
Closed
@paulhauner paulhauner changed the base branch from stable to unstable February 24, 2021 05:06
@paulhauner
Copy link
Member Author

I'm going to run this on 5k Pyrmont validators... Wish me luck :D

@paulhauner
Copy link
Member Author

paulhauner commented Feb 24, 2021 •
edited
Loading

It ran well overnight. It's running on two nodes which are using 3.66gb and 4.20gb (lol) resident memory.

The other non-jemalloc nodes are using between 5.5gb and 6.03gb

@paulhauner
Copy link
Member Author

I'm observing the jemalloc nodes using roughly 2/3rds the resident memory of non-jemalloc nodes. It appears that this PR is quite effective at this point.

paulhauner added a commit to paulhauner/lighthouse that referenced this pull request Mar 30, 2021
@paulhauner
Squashed commit of sigp#2222
954f96f 
commit 13adea78354beb4ff406bd8d5748b560feb37042
Merge: 345802f a764c3b
Author: Paul Hauner <[email protected]>
Date:   Wed Feb 24 16:21:22 2021 +1100

    Merge branch 'unstable' into jemalloc-paul

commit 345802f
Author: Paul Hauner <[email protected]>
Date:   Wed Feb 24 15:58:31 2021 +1100

    Jemalloc by default

commit 88e54ab
Author: Paul Hauner <[email protected]>
Date:   Wed Feb 24 15:41:35 2021 +1100

    Fix config

commit 6680b6d
Merge: 42eb539 46920a8
Author: Paul Hauner <[email protected]>
Date:   Wed Feb 24 14:21:45 2021 +1100

    Merge branch 'unstable' into jemalloc-sea

commit 42eb539
Author: Paul Hauner <[email protected]>
Date:   Tue Feb 16 12:12:34 2021 +1100

    Set jemalloc as default, configure arenas

commit 4331ef8
Merge: d4010e3 4a357c9
Author: Paul Hauner <[email protected]>
Date:   Tue Feb 16 10:00:47 2021 +1100

    Merge branch 'unstable' into jemalloc-sea

commit d4010e3
Author: Paul Hauner <[email protected]>
Date:   Tue Feb 16 10:00:37 2021 +1100

    Make uses jemmaloc

commit 5e968df
Merge: ff14ee0 e2ff9c6
Author: realbigsean <[email protected]>
Date:   Fri Feb 12 10:36:49 2021 -0500

    Merge branch 'unstable' of https://github.com/sigp/lighthouse into jemalloc

    � Conflicts:
    �	Cargo.lock
    �	beacon_node/http_metrics/Cargo.toml
    �	beacon_node/network/src/attestation_service/tests/mod.rs

commit ff14ee0
Author: realbigsean <[email protected]>
Date:   Fri Feb 12 10:35:18 2021 -0500

    add newlines

commit ce1c4e1
Author: realbigsean <[email protected]>
Date:   Fri Feb 12 09:54:19 2021 -0500

    move jemalloc to a feature

commit a0d7119
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 19:12:13 2021 -0500

    cargo lock

commit f205c18
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 18:53:43 2021 -0500

    delete unused dep

commit fd17192
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 18:51:25 2021 -0500

    delete background threads

commit ae8b61a
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 18:50:55 2021 -0500

    fix error handling and remove background threads

commit 9e7a6b7
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 11:48:25 2021 -0500

    semicolon

commit 3408ee1
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 11:46:40 2021 -0500

    update to associated function

commit 074ce1f
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 11:43:25 2021 -0500

    enable background threads

commit 8805ef9
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 10:34:47 2021 -0500

    fix arenas

commit 6f17971
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 10:25:09 2021 -0500

    import arenas

commit 034b677
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 10:23:12 2021 -0500

    add more jemalloc stats

commit 632bff2
Author: realbigsean <[email protected]>
Date:   Wed Feb 10 19:11:30 2021 -0500

    test add jemalloc metrics

commit abf5cac
Author: realbigsean <[email protected]>
Date:   Wed Feb 10 19:03:08 2021 -0500

    test add jemalloc metrics

commit 1d1fc6a
Author: realbigsean <[email protected]>
Date:   Wed Feb 10 18:48:48 2021 -0500

    add jemalloc deps

commit 3c7920b
Author: realbigsean <[email protected]>
Date:   Wed Feb 10 18:34:18 2021 -0500

    test add jemalloc metrics

commit db6fda6
Author: realbigsean <[email protected]>
Date:   Wed Feb 10 18:14:22 2021 -0500

    test add jemalloc
@paulhauner paulhauner mentioned this pull request Mar 30, 2021
Closed
@paulhauner
Copy link
Member Author

Closed in favor of #2288

@paulhauner paulhauner closed this Mar 30, 2021
@paulhauner paulhauner deleted the jemalloc-paul branch March 30, 2021 05:04
paulhauner added a commit to paulhauner/lighthouse that referenced this pull request Mar 30, 2021
@paulhauner
Squashed commit of sigp#2222
6e5d0c4 
commit 13adea78354beb4ff406bd8d5748b560feb37042
Merge: 345802f a764c3b
Author: Paul Hauner <[email protected]>
Date:   Wed Feb 24 16:21:22 2021 +1100

    Merge branch 'unstable' into jemalloc-paul

commit 345802f
Author: Paul Hauner <[email protected]>
Date:   Wed Feb 24 15:58:31 2021 +1100

    Jemalloc by default

commit 88e54ab
Author: Paul Hauner <[email protected]>
Date:   Wed Feb 24 15:41:35 2021 +1100

    Fix config

commit 6680b6d
Merge: 42eb539 46920a8
Author: Paul Hauner <[email protected]>
Date:   Wed Feb 24 14:21:45 2021 +1100

    Merge branch 'unstable' into jemalloc-sea

commit 42eb539
Author: Paul Hauner <[email protected]>
Date:   Tue Feb 16 12:12:34 2021 +1100

    Set jemalloc as default, configure arenas

commit 4331ef8
Merge: d4010e3 4a357c9
Author: Paul Hauner <[email protected]>
Date:   Tue Feb 16 10:00:47 2021 +1100

    Merge branch 'unstable' into jemalloc-sea

commit d4010e3
Author: Paul Hauner <[email protected]>
Date:   Tue Feb 16 10:00:37 2021 +1100

    Make uses jemmaloc

commit 5e968df
Merge: ff14ee0 e2ff9c6
Author: realbigsean <[email protected]>
Date:   Fri Feb 12 10:36:49 2021 -0500

    Merge branch 'unstable' of https://github.com/sigp/lighthouse into jemalloc

    � Conflicts:
    �	Cargo.lock
    �	beacon_node/http_metrics/Cargo.toml
    �	beacon_node/network/src/attestation_service/tests/mod.rs

commit ff14ee0
Author: realbigsean <[email protected]>
Date:   Fri Feb 12 10:35:18 2021 -0500

    add newlines

commit ce1c4e1
Author: realbigsean <[email protected]>
Date:   Fri Feb 12 09:54:19 2021 -0500

    move jemalloc to a feature

commit a0d7119
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 19:12:13 2021 -0500

    cargo lock

commit f205c18
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 18:53:43 2021 -0500

    delete unused dep

commit fd17192
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 18:51:25 2021 -0500

    delete background threads

commit ae8b61a
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 18:50:55 2021 -0500

    fix error handling and remove background threads

commit 9e7a6b7
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 11:48:25 2021 -0500

    semicolon

commit 3408ee1
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 11:46:40 2021 -0500

    update to associated function

commit 074ce1f
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 11:43:25 2021 -0500

    enable background threads

commit 8805ef9
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 10:34:47 2021 -0500

    fix arenas

commit 6f17971
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 10:25:09 2021 -0500

    import arenas

commit 034b677
Author: realbigsean <[email protected]>
Date:   Thu Feb 11 10:23:12 2021 -0500

    add more jemalloc stats

commit 632bff2
Author: realbigsean <[email protected]>
Date:   Wed Feb 10 19:11:30 2021 -0500

    test add jemalloc metrics

commit abf5cac
Author: realbigsean <[email protected]>
Date:   Wed Feb 10 19:03:08 2021 -0500

    test add jemalloc metrics

commit 1d1fc6a
Author: realbigsean <[email protected]>
Date:   Wed Feb 10 18:48:48 2021 -0500

    add jemalloc deps

commit 3c7920b
Author: realbigsean <[email protected]>
Date:   Wed Feb 10 18:34:18 2021 -0500

    test add jemalloc metrics

commit db6fda6
Author: realbigsean <[email protected]>
Date:   Wed Feb 10 18:14:22 2021 -0500

    test add jemalloc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
do-not-merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@paulhauner @realbigsean