[R] CRAN packaging checklist for version 17.0.0 #43317

jonkeane · 2024-07-18T02:05:10Z

Describe the enhancement requested

Packaging checklist for CRAN release

For a high-level overview of the release process see the
Apache Arrow Release Management Guide.

cc @jonkeane @amoeba @nealrichardson @assignUser @paleolimbot for coordination on the various tasks to release

We will want to cherry pick:

Before the release candidate is cut

Create a GitHub issue entitled [R] CRAN packaging checklist for version X.X.X and copy this checklist to the issue.
Review deprecated functions to advance their deprecation status, including removing preprocessor directives that no longer apply (search for ARROW_VERSION_MAJOR in r/src).
Evaluate the status of any failing nightly tests and nightly packaging builds. These checks replicate most of the checks that CRAN runs, so we need them all to be passing or to understand that the failures may (though won't necessarily) result in a rejection from CRAN.
Check current CRAN check results
Ensure the contents of the README are accurate and up to date.
Run urlchecker::url_check() on the R directory at the release candidate.
commit. Ignore any errors with badges as they will be removed in the CRAN release branch.
Polish NEWS but do not update version numbers (this is done automatically later). You can find commits by, for example, git log --oneline aa057d0..HEAD | grep "\[R\]"
Run preliminary reverse dependency checks using archery docker run r-revdepcheck.
For major releases, prepare tweet thread highlighting new features.

Wait for the release candidate to be cut:

After release candidate has been cut

Create a CRAN-release branch from the release candidate commit

Prepare and check the .tar.gz that will be released to CRAN.

git fetch upstream && git checkout release-X.X.X-rcXX && git clean -f -d
Run make build. This copies Arrow C++ into tools/cpp, prunes some
unnecessary components, and runs R CMD build to generate the source tarball.
Because this will install the package, you will need to ensure that the version
of Arrow C++ available to the configure script is the same as the version
that is vendored into the R package (e.g., you may need to unset ARROW_HOME).
devtools::check_built("arrow_X.X.X.tar.gz") locally
Run reverse dependency checks using archery docker run r-revdepcheck.

Release vote

Release vote passed!

Generate R package to submit to CRAN

If the release candidate commit updated, rebase the CRAN release branch
on that commit.
Pick any commits that were made to main since the release commit that
were needed to fix CRAN-related submission issues identified in the above
steps.
Remove badges from README.md
Run urlchecker::url_check() on the R directory
Create a PR entitled WIP: [R] Verify CRAN release-10.0.1-rc0. Add
a comment @github-actions crossbow submit --group r to run all R crossbow
jobs against the CRAN-specific release branch.
Run Rscript tools/update-checksums.R <libarrow version> to download the checksums for the pre-compiled binaries from the ASF artifactory into the tools directory.
Regenerate arrow_X.X.X.tar.gz (i.e., make build)

Ensure linux binary packages are available:

Ensure linux binaries are available in the artifactory:
https://apache.jfrog.io/ui/repos/tree/General/arrow/r

Check binary Arrow C++ distributions specific to the R package

Upload the .tar.gz to win-builder (r-devel only)
and confirm (with Nic, who will automatically receive an email about the results) that the check is clean.
This step cannot be completed before Jeroen has put the binaries in the MinGW repository, i.e. here, here, and here.
Upload the .tar.gz to MacBuilder
and confirm that the check is clean
Check install.packages("arrow_X.X.X.tar.gz") on Ubuntu and ensure that the
hosted binaries are used
devtools::check_built("arrow_X.X.X.tar.gz") locally one more time (for luck)

CRAN submission

Upload arrow_X.X.X.tar.gz to the
CRAN submit page
Confirm the submission email

Wait for CRAN...

Accepted!
Tag the tip of the CRAN-specific release branch
Add a new line to the matrix in the backwards compatability job
(patch releases only) Update the package version in ci/scripts/PKGBUILD, dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb, r/DESCRIPTION, and r/NEWS.md
(CRAN-only releases) Rebuild news page with pkgdown::build_news() and submit a PR to the asf-site branch of the docs site with the contents of arrow/r/docs/news/index.html replacing the current contents of arrow-site/docs/r/news/index.html
(CRAN-only releases) Bump the version number in r/pkgdown/assets/versions.json, and update this on the the asf-site branch of the docs site too.
Update the packaging checklist template to reflect any new realities of the
packaging process.
Wait for CRAN-hosted binaries on the
CRAN package page to reflect the
new version
Tweet!

Component(s)

R

Component(s)

R

The text was updated successfully, but these errors were encountered:

jonkeane · 2024-07-18T02:16:56Z

@assignUser Did you use Rscript tools/update-checksums.R <libarrow version> in the past to generate the contents of 12ec842 ?

assignUser · 2024-07-18T02:17:54Z

Yes :)
Let me know if I can help with anything!

amoeba · 2024-07-18T17:16:50Z

I added the NEWS.md entry from #43189 to the release blog post PR.

assignUser · 2024-07-20T04:15:02Z

@jonkeane mac builder: https://mac.r-project.org/macbuilder/results/1721447405-da61deb72185bfbc/
win builder should reach you by mail, if that's clean I think only a revdep check is missing. @paleolimbot didn't you have a super quick script that doesn't take a million hours to run?

jonkeane · 2024-07-20T18:10:33Z

Thanks for sending those, Jacob.

Here's the winbuilder output: https://win-builder.r-project.org/SrOKYg8yT1lm/

https://win-builder.r-project.org/SrOKYg8yT1lm/00check.log

The only thing in there that looks worrying is

* checking R code for possible problems ... [33s] NOTE
mutate.ArrowTabular: no visible global function definition for
  'left_join'
mutate.Dataset: no visible global function definition for 'left_join'
mutate.RecordBatchReader: no visible global function definition for
  'left_join'
mutate.arrow_dplyr_query: no visible global function definition for
  'left_join'
Undefined global functions or variables:
  left_join
* checking Rd files ... OK

Which I suspect is from https://github.com/apache/arrow/pull/41576/files and can be dealt with by adding dplyr:: onto those calls again

amoeba · 2024-07-20T18:50:18Z

I brought in the changes from maint-17.0.0-r (see my main branch) and kicked off two recheck jobs (1, 2).

Edit: I need to tweak recheck a bit to run so the linked jobs failed right away. I'll update them once recheck is running.
Edit 2: Updated URLs.

nealrichardson · 2024-07-20T20:03:18Z

Which I suspect is from https://github.com/apache/arrow/pull/41576/files and can be dealt with by adding dplyr:: onto those calls again

Sorry! I thought I fixed those but I guess I added more. LMK if you need me to make a PR.

jonkeane · 2024-07-20T20:24:36Z

No problem, I've got one at #43348

I'll merge that if CI shows it gets rid of the note (and there are no other bad consequences) then cherrypick that back to the maintenance branch.

This isn't a here-and-now problem, but boy do I wish we could say "these notes we don't care about but all the others we do" for things like this 🤦

jonkeane · 2024-07-20T20:30:04Z

@amoeba am I reading these logs from recheck correctly that with new arrow parquetize is failing in this example:

> ### Name: csv_to_parquet
> ### Title: Convert a csv or a txt file to parquet format
> ### Aliases: csv_to_parquet
> 
> ### ** Examples
> 
> 
> # Conversion from a local csv file to a single parquet file :
> 
> csv_to_parquet(
+   path_to_file = parquetize_example("region_2022.csv"),
+   path_to_parquet = tempfile(fileext=".parquet")
+ )
Reading data...
Writing data...
Error in Table__from_dots(dots, schema, option_use_threads()) : 
  No Set_elt found for ALTSTRING class [class: vroom_chr, pkg: vroom]
Calls: csv_to_parquet ... as_arrow_table.data.frame -> <Anonymous> -> Table__from_dots
Reading data...
Execution halted

amoeba · 2024-07-20T20:44:56Z

Hey @jonkeane, I think that's the right interpretation.

jonkeane · 2024-07-20T20:49:42Z

Ah, that is a regression on us: #43349 I need to step away for a bit, in case someone wants to investigate what's up there. I don't think much has changed with our altrep code, but this could be from our non-api avoidance changes (or possibly the change we made to avoid executing functions while reading data).

jonkeane · 2024-07-27T16:55:45Z

Ok, I've merged the fix for the altrep thing that was hitting parquetize. @amoeba would you mind triggering the revdep checks again?

jonkeane · 2024-07-30T13:59:40Z

I just got a note about errors from lz4 on CRAN's devel debian runners.

With current Debian testing, installation now fails, see the install log. Apparently this is from yesterday's system upgrade of

liblz4-dev:amd64 (1.9.4-2, 1.9.4-3), liblz4-1:amd64 (1.9.4-2, 1.9.4-3)

I can confirm that ARROW_WITH_LZ4=OFF ./configure works where ./configure now fails.

We will need to fix this as will in our 17 submission (and get it in within two weeks too).

We are linking against the shared, system lz4 which is fine in this case because CRAN does not distribute these binaries. But one possibility is to force non-shared lz4 and it'll grab source + build.

cc @assignUser @nealrichardson

nealrichardson · 2024-07-30T14:14:39Z

We can work around that, sure, but if that's failing, then it sounds like there's something off in our cmake routine in this case, and that should get addressed. For example, I see the message from this line in the logs: https://github.com/apache/arrow/blob/main/cpp/cmake_modules/ThirdpartyToolchain.cmake#L352 But I don't see either of the messages that should follow, about whether lz4 was found or not.

jonkeane · 2024-07-30T14:46:59Z

For example, I see the message from this line in the logs: https://github.com/apache/arrow/blob/main/cpp/cmake_modules/ThirdpartyToolchain.cmake#L352 But I don't see either of the messages that should follow, about whether lz4 was found or not.

When you say you don't see either of the messages that follow are you talking about

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Line 354 in 48782e7

message(STATUS "Using ${RESOLVE_DEPENDENCY_PC_PACKAGE}")

or

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Line 358 in 48782e7

message(STATUS "${RESOLVE_DEPENDENCY_PC_PACKAGE} isn't found")

? If so, that first one (line 354) is actually where the log line that you're seeing is emitted, I believe. The line you linked 352 is adding to the string, but the part that starts with Using ... is emitted at line 354 (so we do get there!)

jonkeane · 2024-07-30T14:47:40Z

@kou might have some ideas + help here too (sorry I should have tagged you on that original comment)

nealrichardson · 2024-07-30T15:01:58Z

Ah my bad, you are right.

kou · 2024-07-31T00:23:21Z

#43468 will solve the problem. I'll complete it soon.

assignUser · 2024-07-31T00:28:36Z

Ah thanks @kou! I was just exploring the lz4 that comes with debian:testing you are right: foreach(_cmake_expected_target IN ITEMS LZ4::lz4_shared LZ4::lz4_static)

jonkeane · 2024-07-31T02:44:42Z

Thank you (again) @kou !

amoeba · 2024-07-31T02:57:13Z

Revdepchecks are running at https://github.com/amoeba/arrow/actions. It looks like one already failed but I don't have enough signal to check right now (will tomorrow).

jonkeane · 2024-07-31T13:33:10Z

FYI, I got another email from CRAN preponing the deadline to ~today since CRAN shuts down for two weeks tomorrow 😬

jonkeane · 2024-07-31T13:38:31Z

~~Weird, I see ^^^ in the logs https://github.com/amoeba/arrow/actions/runs/10172857322/job/28137640795 but when I download the archives I don't see any issues with either of those in the actual logs.~~

Ugh, nevermind, I was looking at the wrong thing! HMMM I see the same issue for parquetize now that I'm looking at the right thing.

Is it possible you haven't pulled the latest updates from https://github.com/apache/arrow/tree/maint-17.0.0-r ?

As for the tidyquery failure, maybe @ianmcook could help figure out what's up with that??

Package: tidyquery 0.2.4
Check: tests, Result: ERROR
    Running ‘testthat.R’
  Running the tests in ‘tests/testthat.R’ failed.
  Last 13 lines of output:
      6. ├─tidyquery::query("SELECT origin, dest,\n          COUNT(flight) AS num_flts,\n          round(SUM(seats)) AS num_seats,\n          round(AVG(arr_delay)) AS avg_delay\n      FROM flights_at f LEFT OUTER JOIN planes_at p\n        ON f.tailnum = p.tailnum\n      WHERE distance BETWEEN 200 AND 300\n      AND air_time IS NOT NULL\n      GROUP BY origin, dest\n      -- HAVING num_flts > 3000\n      ORDER BY num_seats DESC, avg_delay ASC\n      LIMIT 100;")
      7. │ └─tidyquery:::query_(data, sql, TRUE)
      8. │   └─out %>% verb(filter, !!(tree$where[[1]]))
      9. ├─tidyquery:::verb(., filter, !!(tree$where[[1]]))
     10. │ └─input$data %>% fun(...)
     11. ├─dplyr (local) fun(., ...)
     12. └─arrow:::filter.arrow_dplyr_query(., ...)
     13.   ├─arrow:::try_arrow_dplyr(...)
     14.   │ └─base::evalq(call <- match.call(), parent)
     15.   │   └─base::evalq(call <- match.call(), parent)
     16.   └─base::match.call()
    
    [ FAIL 9 | WARN 0 | SKIP 4 | PASS 226 ]
    Error: Test failures
    Execution halted

jonkeane · 2024-07-31T14:14:06Z

I've also tried to replicate the CRAN devel image locally (I'll push a PR up with that for CI after this fire drill). After Kou's change, we get past LZ4. But then it seems to fail with the following. This isn't related directly to the LZ4 change I don't think, and as far as I could tell on github that abseil code hasn't changed in years. This could be a false positive, since I don't have an exact replica of the CRAN runner, but wonder if anyone has any ideas about what this might be.

[ 55%] Building CXX object absl/strings/CMakeFiles/str_format_internal.dir/internal/str_format/extension.cc.o

-- stderr output is:
...skipping to end...
230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:45:1: note: in expansion of macro ‘ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_’
   45 | ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_(ABSL_INTERNAL_CHAR_SET_CASE, )
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:44:13: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   44 |   constexpr FormatConversionCharSet FormatConversionCharSetInternal::c;
      |             ^~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.h:167:3: note: in expansion of macro ‘ABSL_INTERNAL_CHAR_SET_CASE’
  167 |   X_VAL(g) X_SEP X_VAL(G) X_SEP X_VAL(a) X_SEP X_VAL(A) X_SEP \
      |   ^~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:45:1: note: in expansion of macro ‘ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_’
   45 | ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_(ABSL_INTERNAL_CHAR_SET_CASE, )
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:44:13: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   44 |   constexpr FormatConversionCharSet FormatConversionCharSetInternal::c;
      |             ^~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.h:167:18: note: in expansion of macro ‘ABSL_INTERNAL_CHAR_SET_CASE’
  167 |   X_VAL(g) X_SEP X_VAL(G) X_SEP X_VAL(a) X_SEP X_VAL(A) X_SEP \
      |                  ^~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:45:1: note: in expansion of macro ‘ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_’
   45 | ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_(ABSL_INTERNAL_CHAR_SET_CASE, )
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:44:13: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   44 |   constexpr FormatConversionCharSet FormatConversionCharSetInternal::c;
      |             ^~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.h:167:33: note: in expansion of macro ‘ABSL_INTERNAL_CHAR_SET_CASE’
  167 |   X_VAL(g) X_SEP X_VAL(G) X_SEP X_VAL(a) X_SEP X_VAL(A) X_SEP \
      |                                 ^~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:45:1: note: in expansion of macro ‘ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_’
   45 | ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_(ABSL_INTERNAL_CHAR_SET_CASE, )
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:44:13: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   44 |   constexpr FormatConversionCharSet FormatConversionCharSetInternal::c;
      |             ^~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.h:167:48: note: in expansion of macro ‘ABSL_INTERNAL_CHAR_SET_CASE’
  167 |   X_VAL(g) X_SEP X_VAL(G) X_SEP X_VAL(a) X_SEP X_VAL(A) X_SEP \
      |                                                ^~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:45:1: note: in expansion of macro ‘ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_’
   45 | ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_(ABSL_INTERNAL_CHAR_SET_CASE, )
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:44:13: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   44 |   constexpr FormatConversionCharSet FormatConversionCharSetInternal::c;
      |             ^~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.h:169:3: note: in expansion of macro ‘ABSL_INTERNAL_CHAR_SET_CASE’
  169 |   X_VAL(n) X_SEP X_VAL(p)
      |   ^~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:45:1: note: in expansion of macro ‘ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_’
   45 | ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_(ABSL_INTERNAL_CHAR_SET_CASE, )
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:44:13: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   44 |   constexpr FormatConversionCharSet FormatConversionCharSetInternal::c;
      |             ^~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.h:169:18: note: in expansion of macro ‘ABSL_INTERNAL_CHAR_SET_CASE’
  169 |   X_VAL(n) X_SEP X_VAL(p)
      |                  ^~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:45:1: note: in expansion of macro ‘ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_’
   45 | ABSL_INTERNAL_CONVERSION_CHARS_EXPAND_(ABSL_INTERNAL_CHAR_SET_CASE, )
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:49:11: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   49 | constexpr FormatConversionCharSet FormatConversionCharSetInternal::kStar;
      |           ^~~~~~~~~~~~~~~~~~~~~~~
      |           FormatConversionCharInternal
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:51:11: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   51 | constexpr FormatConversionCharSet FormatConversionCharSetInternal::kIntegral;
      |           ^~~~~~~~~~~~~~~~~~~~~~~
      |           FormatConversionCharInternal
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:53:11: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   53 | constexpr FormatConversionCharSet FormatConversionCharSetInternal::kFloating;
      |           ^~~~~~~~~~~~~~~~~~~~~~~
      |           FormatConversionCharInternal
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:55:11: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   55 | constexpr FormatConversionCharSet FormatConversionCharSetInternal::kNumeric;
      |           ^~~~~~~~~~~~~~~~~~~~~~~
      |           FormatConversionCharInternal
/tmp/RtmpabvneB/file7a47e230e44/absl_ep-prefix/src/absl_ep/absl/strings/internal/str_format/extension.cc:57:11: error: ‘FormatConversionCharSet’ does not name a type; did you mean ‘FormatConversionCharInternal’?
   57 | constexpr FormatConversionCharSet FormatConversionCharSetInternal::kPointer;
      |           ^~~~~~~~~~~~~~~~~~~~~~~
      |           FormatConversionCharInternal
make[5]: *** [absl/strings/CMakeFiles/str_format_internal.dir/build.make:104: absl/strings/CMakeFiles/str_format_internal.dir/internal/str_format/extension.cc.o] Error 1
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [CMakeFiles/Makefile2:2382: absl/strings/CMakeFiles/str_format_internal.dir/all] Error 2
make[3]: *** [Makefile:136: all] Error 2

amoeba · 2024-07-31T16:57:53Z

Whoops, sorry about that @jonkeane. New revdep check runs are now here: strong, most. Those are being run from the latest maint-17.0.0-r. They seem to be stuck at the moment so hopefully they'll start soon.

amoeba · 2024-07-31T17:04:32Z

#36969 looks similar to your above output, maybe this is the same cause?

jonkeane · 2024-07-31T19:23:12Z

@assignUser Do you know if we will need to re-generate the Apache-hosted binaries to include #43468 and #43157 and use those in this CRAN release? Or are we ok without those in the binaries since those are both build tools only?

jonkeane · 2024-07-31T19:40:57Z

Looks like tidyquery is still an issue (I'm surprised we didn't catch this the first time we ran through those!)

I'm seeing a bunch of errors like this one. Which I suspect are coming from

arrow/r/R/dplyr-eval.R

Line 204 in 7e50097

evalq(call <- match.call(), parent)

✖ | 3    1   1 | arrow-dataset                                                                    
──────────────────────────────────────────────────────────────────────────────────────────────────
Error (test-arrow-dataset.R:3:3): Simple SELECT example query #1 returns expected result on Arrow Dataset
Error in `match.call()`: ... used in a situation where it does not exist
Backtrace:
     ▆
  1. ├─testthat::expect_equal(...) at test-arrow-dataset.R:3:3
  2. │ └─testthat::quasi_label(enquo(object), label, arg = "object")
  3. │   └─rlang::eval_bare(expr, quo_get_env(quo))
  4. ├─... %>% as.data.frame()
  5. ├─base::as.data.frame(.)
  6. ├─tidyquery::query("SELECT Species, COUNT(*) AS n FROM iris_ad GROUP BY Species")
  7. │ └─tidyquery:::query_(data, sql, TRUE) at tidyquery-9546a7d191009cc56c2c11c5a54c33d3330a42f3/R/query.R:77:3
  8. │   └─... %>% verb(ungroup) at tidyquery-9546a7d191009cc56c2c11c5a54c33d3330a42f3/R/query.R:275:7
  9. ├─tidyquery:::verb(., ungroup)
 10. │ └─base::paste(...) at tidyquery-9546a7d191009cc56c2c11c5a54c33d3330a42f3/R/query.R:450:3
 11. ├─tidyquery:::verb(...)
 12. │ └─input$data %>% fun(...) at tidyquery-9546a7d191009cc56c2c11c5a54c33d3330a42f3/R/query.R:450:3
 13. ├─dplyr (local) fun(., ...)
 14. └─arrow:::summarise.arrow_dplyr_query(., ...)
 15.   ├─arrow:::try_arrow_dplyr(...)
 16.   │ └─base::evalq(call <- match.call(), parent)
 17.   │   └─base::evalq(call <- match.call(), parent)
 18.   └─base::match.call()

### Rationale for this change See #43317 (comment). `tidyquery` is assembling queries in some way such that when `summarize.arrow_dplyr_query()` is called, the calling environment isn't a call, so `match.call()` fails. ### What changes are included in this PR? This PR wraps the `match.call()` call in a `try()`. The call is only used to do `abandon_ship()` on in-memory data anyway. So if the call is not available, it treats it like you're making a query on a Dataset and it tells you to `collect()` yourself. ### Are these changes tested? I couldn't figure out how to reproduce what was going on inside `tidyquery` to write a reproducer, and I don't think this is worth adding `tidyquery` to Suggests for. I confirmed locally that `tidyquery` tests pass with this change, so our revdeps should be clear. ### Are there any user-facing changes? 🙅 Authored-by: Neal Richardson <[email protected]> Signed-off-by: Jonathan Keane <[email protected]>

jonkeane · 2024-07-31T23:17:29Z

Thanks for all of the help, everyone. I just sent the submission, I'll keep this issue posted if there are other issues.

jonkeane · 2024-08-01T00:53:29Z

Womp womp. I received the not-passing-auto-checks email. The problem is that we are calling XXX on <4.2 and CRAN's checks for these apparently are static and not actually checking if we are truly calling them (on R versions that they run on).

I'm partially tempted to wait and see if they end up accepting it anyway after seeing what's going on. I could also reply and explain (though I have extremely low hopes of that landing, even though it is simply a fact that before 4.2 the approved thing does not exist!). @nealrichardson do I remember you having an idea on possibly obfuscating this call for our compatibility branch?

The only NOTE is:

* checking compiled code ... NOTE
File 'arrow/libs/x64/arrow.dll':
  Found non-API calls to R: 'Rf_findVarInFrame3', 'SETLENGTH',
    'SET_GROWABLE_BIT', 'SET_TRUELENGTH'

Compiled code should not call non-API entry points in R.

See 'Writing portable packages' in the 'Writing R Extensions' manual,
and section 'Moving into C API compliance' for issues with the use of
non-API entry points.

https://win-builder.r-project.org/incoming_pretest/arrow_17.0.0_20240801_011430/Windows/00check.log
https://win-builder.r-project.org/incoming_pretest/arrow_17.0.0_20240801_011430/Debian/00check.log

nealrichardson · 2024-08-01T01:47:26Z

You could wait to see if Uwe waves it through, or you could reply and explain that these are ifdef'd for backwards compatibility only.

The in-case-of-emergency hack would be to sed out these references from the .cpp files in the configure script if R is > 4.4 (the checks are only on devel now, right?). But it seems like this is a more general problem CRAN will have to solve, if they're adding new "official" API functions to replace some of these things but they're only in R-devel, how can you be on CRAN if you depend on them?

What's surprising to me is that, at least by the internal function names check is calling, they're looking at the compiled object files, which I would expect not to have these symbols in them because of the #ifdef: https://github.com/wch/r-source/blob/032b8240be6c8cf5e8e85ecde6f3f23e46739716/src/library/tools/R/sotools.R#L721-L768 And that function is wrapping nm.

In fact, I do see these symbols showing up locally, on new enough R, for example:

npr@nealrichardson-3P29 r % nm src/arrow.so | grep Rf_findVarInFrame3
                 U _Rf_findVarInFrame3

nealrichardson · 2024-08-01T01:48:36Z

Of course CRAN is closed starting tomorrow (and it's already tomorrow CEST), so this is probably not something we can solve until they're back.

jonkeane · 2024-08-01T04:27:24Z

Hmmmm, yeah something strange is going on here. It's possible that the #IFDEF isn't the right thing for detecting functions (https://stackoverflow.com/questions/6234755/does-ifdef-or-other-preprocessor-directives-work-for-function-declarations-t) and instead we should do something like #if (R_VERSION >= R_Version(4, 2, 0)) buuuuut I actually still see _Rf_findvarInFrame3 even if I remove any calls (conditional or otherwise) from arrow_cpp11.h (which is also the only place that calls it in our code) 🫠 I do see it in cpp11 https://github.com/r-lib/cpp11/blob/1c9dbb65cc9279a404f035f2fca990b338c9f602/inst/include/cpp11/environment.hpp#L37-L38 so maybe that's where it's coming from?

jonkeane · 2024-08-01T04:50:53Z

Ah yes, I think it is, indeed, coming from cpp11's environment.hpp. I just tried removing all of our imports / references to that and only then is nm -v src/arrow.so | grep Rf_findVarInFrame3 clean.

We still need #43504 but will still have an issue on that for automatic checks until cpp11 fixes + releases their fix.

nealrichardson · 2024-08-01T11:59:56Z

We may need to vendor (and possibly patch) cpp11 for this release then.

nealrichardson · 2024-08-14T14:51:45Z

Need to cherry-pick #43649 (ab432b1) into the CRAN release

jonkeane · 2024-08-18T14:26:42Z

Ok, I've cherry-picked the latest fix onto this branch (#43735). I'll resubmit later this evening — in case anyone has any objections + suggestions for other things.

jonkeane · 2024-08-19T00:29:31Z

I have resubmitted 17.0.0.1

jonkeane · 2024-08-21T20:34:53Z

Aaaand we have been accepted 🎉

Do we already have a draft social media post?

amoeba · 2024-08-22T00:24:36Z

I haven't seen a draft yet. @thisisnic do you want to take this on again?

thisisnic · 2024-08-22T07:09:17Z

I was taking this release "off" to focus on my other commitments, though would be happy to review it if someone else wanted to write something up?

amoeba · 2024-08-22T16:51:34Z

I can get it started later today, I'll drop a link in here for reviews when that's done.

amoeba · 2024-08-23T18:00:27Z

Draft social media content is here. I still need to generate some code images for the examples and tweak a bit of the language but please have a look to see if I missed any tweet-worthy highlights. I see binaries are almost all up on CRAN which is great.

thisisnic · 2024-08-23T18:26:51Z

Language here looks great, thanks @amoeba!

jonkeane · 2024-08-23T18:51:40Z

Thanks, this is really great! I'll also post it on linked in (though not as a series). I also took the liberty to add emoji in the places to make it engaging 😂

assignUser · 2024-08-23T22:20:35Z

We are missing a checkbox to move the r-univerese tag, I'll move the tag forward in a bit
e: done

amoeba · 2024-08-26T04:33:53Z

Thanks @jonkeane and @thisisnic for taking a look at the social media doc. I accepted suggestions and added code screenshots to the Drive folder where appropriate, see https://drive.google.com/drive/folders/1DzGoZNHbiZXNca2oHWjpBtJrEti3W-zQ?usp=drive_link, so I think this is ready for me and @jonkeane to post when x86_64 binaries go live.

amoeba · 2024-09-12T16:04:43Z

It looks like the macOS x86_64 binary finally got built, see https://cran.r-project.org/web/packages/arrow/index.html. The binary for oldrel still isn't built but I don't think that should prevent any announcement about the release. Thoughts?

thisisnic · 2024-09-14T19:31:47Z

Let's do it! Thanks @amoeba!

jonkeane · 2024-09-15T12:53:45Z

Agreed, let's call it. The oldrelease is one that I've asked about before on mailing lists and they sometimes are super slow, but I've seen periods where they don't build for any packages for weeks and eventually sort themselves.

I can RTFS, but does anyone know how / if we can update the docs website to show the 17.0.0.1 number?

amoeba · 2024-09-17T04:22:47Z

I scheduled tweets/toots to go out tomorrow morning.

Re: The website showing version 17.0.0... I think we might want to find a better workflow but for now I created apache/arrow-site#542 to do the update. My steps are described there.

Edit: Threads up at https://x.com/brycem/status/1836040825752490139 and https://elk.zone/toot.cafe/@amoeba/113153321269054346

jonkeane · 2024-09-22T13:41:28Z

I've also made an article on linkedin not sure if that's the right level of linked in posting, but I'll keep an eye on if it gets engagement.

Thanks for the help, everyone! I'm going to close this out.

amoeba · 2024-09-23T15:44:27Z

Thanks @jonkeane. Do we need to update the r-universe-release tag still and add a r-17.0.01 tag?

jonkeane · 2024-09-23T17:23:08Z

Yeah, if we don't have one already we should. Sorry I didn't look at that

jonkeane added the Type: enhancement label Jul 18, 2024

jonkeane self-assigned this Jul 18, 2024

amoeba mentioned this issue Aug 3, 2024

[R][CI] Add r-devel/recheck workflow for doing reverse dependency checks on CI #43547

Open

3 tasks

amoeba added the Component: R label Sep 12, 2024

jonkeane closed this as completed Sep 22, 2024

[R] CRAN packaging checklist for version 17.0.0 #43317

[R] CRAN packaging checklist for version 17.0.0 #43317

Comments

jonkeane commented Jul 18, 2024 • edited by amoeba Loading

Describe the enhancement requested

Packaging checklist for CRAN release

Before the release candidate is cut

After release candidate has been cut

Prepare and check the .tar.gz that will be released to CRAN.

Release vote

Generate R package to submit to CRAN

Check binary Arrow C++ distributions specific to the R package

CRAN submission

Component(s)

Component(s)

jonkeane commented Jul 18, 2024

assignUser commented Jul 18, 2024

amoeba commented Jul 18, 2024

assignUser commented Jul 20, 2024

jonkeane commented Jul 20, 2024

amoeba commented Jul 20, 2024 • edited Loading

nealrichardson commented Jul 20, 2024

jonkeane commented Jul 20, 2024 • edited Loading

jonkeane commented Jul 20, 2024

amoeba commented Jul 20, 2024

jonkeane commented Jul 20, 2024

jonkeane commented Jul 27, 2024

jonkeane commented Jul 30, 2024

nealrichardson commented Jul 30, 2024

jonkeane commented Jul 30, 2024

jonkeane commented Jul 30, 2024

nealrichardson commented Jul 30, 2024

kou commented Jul 31, 2024

assignUser commented Jul 31, 2024

jonkeane commented Jul 31, 2024

amoeba commented Jul 31, 2024

jonkeane commented Jul 31, 2024

jonkeane commented Jul 31, 2024 • edited Loading

jonkeane commented Jul 31, 2024

amoeba commented Jul 31, 2024

amoeba commented Jul 31, 2024

jonkeane commented Jul 31, 2024

jonkeane commented Jul 31, 2024

jonkeane commented Jul 31, 2024

jonkeane commented Aug 1, 2024

nealrichardson commented Aug 1, 2024

nealrichardson commented Aug 1, 2024

jonkeane commented Aug 1, 2024

jonkeane commented Aug 1, 2024 • edited Loading

nealrichardson commented Aug 1, 2024

nealrichardson commented Aug 14, 2024 • edited Loading

jonkeane commented Aug 18, 2024

jonkeane commented Aug 19, 2024

jonkeane commented Aug 21, 2024

amoeba commented Aug 22, 2024

thisisnic commented Aug 22, 2024

amoeba commented Aug 22, 2024

amoeba commented Aug 23, 2024

thisisnic commented Aug 23, 2024

jonkeane commented Aug 23, 2024 • edited Loading

assignUser commented Aug 23, 2024 • edited Loading

amoeba commented Aug 26, 2024

amoeba commented Sep 12, 2024

thisisnic commented Sep 14, 2024

jonkeane commented Sep 15, 2024

amoeba commented Sep 17, 2024 • edited Loading

jonkeane commented Sep 22, 2024

amoeba commented Sep 23, 2024

jonkeane commented Sep 23, 2024

jonkeane commented Jul 18, 2024 •

edited by amoeba

Loading

amoeba commented Jul 20, 2024 •

edited

Loading

jonkeane commented Jul 20, 2024 •

edited

Loading

jonkeane commented Jul 31, 2024 •

edited

Loading

jonkeane commented Aug 1, 2024 •

edited

Loading

nealrichardson commented Aug 14, 2024 •

edited

Loading

jonkeane commented Aug 23, 2024 •

edited

Loading

assignUser commented Aug 23, 2024 •

edited

Loading

amoeba commented Sep 17, 2024 •

edited

Loading