[help] "object of type 'symbol' is not subsettable" error when trying to use crew with remote hosts

[billdenney]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

When I run the following

library(targets)

tar_option_set(controller = crew::crew_controller(client = crew::crew_client(workers = 4, port = 5555)))
list(
  tar_target(
    one,
    1
  )
)

I get the following error:

> tar_make()
Error:
! Error running targets::tar_make()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: object of type 'symbol' is not subsettable
> targets::tar_meta(fields = error, complete_only = TRUE)
# A tibble: 0 × 2
# ℹ 2 variables: name <chr>, error <lgl>

When I omit the tar_options_set() line, I don't get any error.

I'm running on Windows 11 with nanonext_0.9.0.9010, mirai_0.8.7.9029, crew_0.2.1.9012, and targets_1.1.3.

Do you have any advice on what I'm doing wrong with my crew setup? I have confirmed that I can connect to the remote server with mirai alone.

[billdenney]

FYI, I updated targets to 1.1.3.9000, and I'm getting the same error.

[wlandau]

Would you try with tar_option_set(controller = crew::crew_controller_local(workers = 4))? This defines both the client and the launcher inside the controller. Also, most cases do not require specifying a port.

[wlandau]

crew_controller() is not really a user-side function, it is for developers of launcher plugins. wlandau/crew@09ab634 makes this clearer in the documentation website of crew and also improves the error message generated by crew_controller().

[billdenney]

Using tar_option_set(controller = crew::crew_controller_local(workers = 4)) fixed the error message, but I don't think that it did what I was hoping for it to do.

I'm trying to enable crew to work from my local Windows laptop to send to a separate compute server. That was the reason I was using a port-- so that I knew how to attach from the compute server back to the mirai daemon. Apologies if I'm missing it, but is there documentation on how to connect back from a separate computer back to the targets main computer?

I just tried crew::crew_controller_local(workers = 4, port = 5555) which created a mirai client with 4 different web sockets paths that also included what appears to be a random hash at the end. I don't know how I would be able to connect back from the compute server to that web socket path since it appears to be dynamically created. (Or maybe, I don't know how web sockets work well enough.)

[wlandau]

In that case, the crew workers need to be started on that separate server. The way that workers get launched depends on the launcher object in the crew controller. crew ships with a "local" launcher which starts local processes, and crew.cluster starts workers on cluster schedules like SLURM and PBS/TORQUE. For your situation, I recommend writing a launcher plugin that starts R processes on the specific compute environment where you need them to run. The process is straightforward, and the vignette at https://wlandau.github.io/crew/articles/plugins.html walks through it.

[billdenney]

Thanks for that answer. I'll give it a shot.