Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Background execution and R process pool #311

Merged
merged 79 commits into from
Aug 9, 2023
Merged

Background execution and R process pool #311

merged 79 commits into from
Aug 9, 2023

Commits on Jun 25, 2023

  1. Prototype execution in background thread

    @Sicheng-Pan
    Sicheng-Pan committed Jun 25, 2023
    Configuration menu
    Copy the full SHA
    d577bc4 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2023

  1. Refactor rthreadhandle prototype

    @Sicheng-Pan
    Sicheng-Pan committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    9918a64 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2023

  1. Prototype background R process

    @Sicheng-Pan
    Sicheng-Pan committed Jul 2, 2023
    Configuration menu
    Copy the full SHA
    cb41db2 View commit details
    Browse the repository at this point in the history

  2. Made a functional R subprocess!

    @Sicheng-Pan
    Sicheng-Pan committed Jul 2, 2023
    Configuration menu
    Copy the full SHA
    eae0ebc View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2023

  1. Refactor code for background handler

    @Sicheng-Pan
    Sicheng-Pan committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    9b1ad25 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2023

  1. Implement R process pool

    @Sicheng-Pan
    Sicheng-Pan committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    18f98c1 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into lazy_in_background

    @Sicheng-Pan
    Sicheng-Pan committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    f907a22 View commit details
    Browse the repository at this point in the history

  3. Attempt to (de)serialize series from polars

    @Sicheng-Pan
    Sicheng-Pan committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    fdee6b5 View commit details
    Browse the repository at this point in the history

  4. Rename R background module file

    @Sicheng-Pan
    Sicheng-Pan committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    4f5886b View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. Implement expr and lf

    @Sicheng-Pan
    Sicheng-Pan committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    fd3a9fb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into lazy_in_background

    @Sicheng-Pan
    Sicheng-Pan committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    4c8e06c View commit details
    Browse the repository at this point in the history

  3. Implement R wrappers for thread handle and global R pool config

    @Sicheng-Pan
    Sicheng-Pan committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    41f180a View commit details
    Browse the repository at this point in the history

  4. Use private function to handle request in subprocess

    @Sicheng-Pan
    Sicheng-Pan committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    020f902 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. Fix handle display

    @Sicheng-Pan
    Sicheng-Pan committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    513fee6 View commit details
    Browse the repository at this point in the history

  2. Fix error handling in background process

    @Sicheng-Pan
    Sicheng-Pan committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    717ecd7 View commit details
    Browse the repository at this point in the history

  3. Fix more error handling issues

    @Sicheng-Pan
    Sicheng-Pan committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    44799fc View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2023

  1. Use shared memory for IPC communication

    @Sicheng-Pan
    Sicheng-Pan committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    5b9ffed View commit details
    Browse the repository at this point in the history

  2. make build export NOT_CRAN=true

    @sorhawell
    sorhawell committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    6452598 View commit details
    Browse the repository at this point in the history

  3. functions to load and check polars in an R session

    @sorhawell
    sorhawell committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    c60e596 View commit details
    Browse the repository at this point in the history

  4. now it work

    @sorhawell
    sorhawell committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    49181f7 View commit details
    Browse the repository at this point in the history

  5. update tip to readme

    @sorhawell
    sorhawell committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    ecd7199 View commit details
    Browse the repository at this point in the history

  6. add test_serde_df add benchmark script

    @sorhawell
    sorhawell committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    0100441 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into lazy_in_background

    @Sicheng-Pan
    Sicheng-Pan committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    c68a443 View commit details
    Browse the repository at this point in the history

  8. Minor refactor

    @Sicheng-Pan
    Sicheng-Pan committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    3c1e045 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2023

  1. rename

    @sorhawell
    sorhawell committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    07d150d View commit details
    Browse the repository at this point in the history

  2. add build and submit_polars + fix minors

    @sorhawell
    sorhawell committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    397c4dd View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2023

  1. Merge remote-tracking branch 'origin/not_cran_dev_builds' into lazy_i… 

    …n_background
    @Sicheng-Pan
    Sicheng-Pan committed Jul 15, 2023
    Configuration menu
    Copy the full SHA
    196e548 View commit details
    Browse the repository at this point in the history

  2. Implement apply-in-background

    @Sicheng-Pan
    Sicheng-Pan committed Jul 15, 2023
    Configuration menu
    Copy the full SHA
    4557b1b View commit details
    Browse the repository at this point in the history

  3. Write unit tests

    @Sicheng-Pan
    Sicheng-Pan committed Jul 15, 2023
    Configuration menu
    Copy the full SHA
    daa90c8 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2023

  1. Benchmark rbackground

    @Sicheng-Pan
    Sicheng-Pan committed Jul 16, 2023
    Configuration menu
    Copy the full SHA
    78a03c8 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2023

  1. add expectations print + is_finished handle

    @sorhawell
    sorhawell committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    d990822 View commit details
    Browse the repository at this point in the history

  2. add more scenarios

    @sorhawell
    sorhawell committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    7a02517 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2023

  1. Improve display of joined handle

    @Sicheng-Pan
    Sicheng-Pan committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    f4b1f0f View commit details
    Browse the repository at this point in the history

  2. RThreadHandle_is_finished.Rd

    @sorhawell
    sorhawell committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    57def1f View commit details
    Browse the repository at this point in the history

  3. try remove NOT_CRAN

    @sorhawell
    sorhawell committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    3fee2c5 View commit details
    Browse the repository at this point in the history

  4. try quote R -e arg

    @sorhawell
    sorhawell committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    d0758f0 View commit details
    Browse the repository at this point in the history

  5. remove R native pipe operator

    @sorhawell
    sorhawell committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    3cbaceb View commit details
    Browse the repository at this point in the history

  6. try not use arg method

    @sorhawell
    sorhawell committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    5f05fce View commit details
    Browse the repository at this point in the history

  7. write failed cmd string in error

    @sorhawell
    sorhawell committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    0694664 View commit details
    Browse the repository at this point in the history

  8. drop cmd_string

    @sorhawell
    sorhawell committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    4ef435e View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2023

  1. try add line ending

    @sorhawell
    sorhawell committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    c7653c2 View commit details
    Browse the repository at this point in the history

  2. try not redirect std

    @sorhawell
    sorhawell committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    88d8b52 View commit details
    Browse the repository at this point in the history

  3. dunno

    @sorhawell
    sorhawell committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    88048f1 View commit details
    Browse the repository at this point in the history

  4. Revert "RThreadHandle_is_finished.Rd" 

    This reverts commit 57def1f.
    @sorhawell
    sorhawell committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    3d81fcf View commit details
    Browse the repository at this point in the history

  5. revert experiments

    @sorhawell
    sorhawell committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    0781242 View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2023

  1. Try std::env::current_exe

    @Sicheng-Pan
    Sicheng-Pan committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    65bb48f View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2023

  1. Merge remote-tracking branch 'origin/main' into lazy_in_background

    @Sicheng-Pan
    Sicheng-Pan committed Jul 23, 2023
    Configuration menu
    Copy the full SHA
    ca17fa3 View commit details
    Browse the repository at this point in the history

  2. Update extendr

    @Sicheng-Pan
    Sicheng-Pan committed Jul 23, 2023
    Configuration menu
    Copy the full SHA
    b03de39 View commit details
    Browse the repository at this point in the history

  3. Try environment variable

    @Sicheng-Pan
    Sicheng-Pan committed Jul 23, 2023
    Configuration menu
    Copy the full SHA
    00c2b4d View commit details
    Browse the repository at this point in the history

  4. change low io high cpu example

    @sorhawell
    sorhawell committed Jul 23, 2023
    Configuration menu
    Copy the full SHA
    97b2fb9 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2023

  1. update benchmark

    @sorhawell
    sorhawell committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    980abe8 View commit details
    Browse the repository at this point in the history

  2. improve Expr_map doc

    @sorhawell
    sorhawell committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    bd15917 View commit details
    Browse the repository at this point in the history

  3. collect_in_background docs

    @sorhawell
    sorhawell committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    d21a881 View commit details
    Browse the repository at this point in the history

  4. update oxygen

    @sorhawell
    sorhawell committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    75d0252 View commit details
    Browse the repository at this point in the history

  5. super minor, Rctx::Handled

    @sorhawell
    sorhawell committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    c2b6237 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2023

  1. Track R environment in Rust

    @Sicheng-Pan
    Sicheng-Pan committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    0dee41a View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2023

  1. Remove old PolarsBackgroundHandler

    @Sicheng-Pan
    Sicheng-Pan committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    d4dbb5f View commit details
    Browse the repository at this point in the history

  2. chore rextendr 0.3.1.9000 nanoarrow .Rd

    @sorhawell
    sorhawell committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    d9e2dab View commit details
    Browse the repository at this point in the history

  3. avoid Rctx.into() -> RpolarsErr

    @sorhawell
    sorhawell committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    388297a View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2023

  1. Merge apply_in_background to apply

    @Sicheng-Pan
    Sicheng-Pan committed Jul 30, 2023
    Configuration menu
    Copy the full SHA
    87adf0d View commit details
    Browse the repository at this point in the history

  2. Improve background error handling

    @Sicheng-Pan
    Sicheng-Pan committed Jul 30, 2023
    Configuration menu
    Copy the full SHA
    a2c30ae View commit details
    Browse the repository at this point in the history

  3. make: redefine install, add install to docs

    @sorhawell
    sorhawell committed Jul 30, 2023
    Configuration menu
    Copy the full SHA
    13bd3c7 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2023

  1. merge main 

    Merge branch 'main' into lazy_in_background
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
    @sorhawell
    sorhawell committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    d0f1765 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into lazy_in_background

    @sorhawell
    sorhawell committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    ecd6a7c View commit details
    Browse the repository at this point in the history

  3. more docs + fix RBackgroundPool

    @sorhawell
    sorhawell committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    bc4bb4c View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2023

  1. impl thread queue in RBackGroundPool

    @sorhawell
    sorhawell committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    08d8eab View commit details
    Browse the repository at this point in the history

  2. spawn processes in paralell

    @sorhawell
    sorhawell committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    4fea62d View commit details
    Browse the repository at this point in the history

  3. add test 3d to benchmark

    @sorhawell
    sorhawell committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    2777c94 View commit details
    Browse the repository at this point in the history

  4. increase default pool size to 4

    @sorhawell
    sorhawell committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    95122a5 View commit details
    Browse the repository at this point in the history

  5. update unit test

    @sorhawell
    sorhawell committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    bc12766 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2023

  1. add parallel examples to map and apply

    @sorhawell
    sorhawell committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    1cdefe4 View commit details
    Browse the repository at this point in the history

  2. merge main + update docs 

    Merge branch 'main' into lazy_in_background

# Conflicts:
#	R/extendr-wrappers.R
#	src/rust/src/lazy/dsl.rs
    @sorhawell
    sorhawell committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    1616095 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2023

  1. merge main + update news + roxygen 

    Merge branch 'main' into lazy_in_background

# Conflicts:
#	man/pl_pl.Rd
#	src/rust/src/lazy/dsl.rs
#	src/rust/src/rpolarserr.rs
    @sorhawell
    sorhawell committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    90d8be7 View commit details
    Browse the repository at this point in the history

  2. polish news [skip ci]

    @sorhawell
    sorhawell committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    afee8a9 View commit details
    Browse the repository at this point in the history

  3. add links

    @sorhawell
    sorhawell committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    61cc469 View commit details
    Browse the repository at this point in the history

  4. try rename all-features to full features

    @sorhawell
    sorhawell committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    8af927f View commit details
    Browse the repository at this point in the history

  5. fmt

    @sorhawell
    sorhawell committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    46058db View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2023

  1. merge main + solve conflicts + docs + fix a utests 

    Merge branch 'main' into lazy_in_background

# Conflicts:
#	NEWS.md
#	R/expr__expr.R
#	R/extendr-wrappers.R
#	R/lazyframe__lazy.R
#	man/LazyFrame_collect_background.Rd
#	src/rust/src/lazy/dataframe.rs
    @sorhawell
    sorhawell committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    f352e0b View commit details
    Browse the repository at this point in the history

  2. fmt + roxygen

    @sorhawell
    sorhawell committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    b6916e4 View commit details
    Browse the repository at this point in the history