Packages (xlsx,dplyr) - Execution in HPC Cluster #314
-
|
Is there any way to prepare an excelsheet rows with xlsx package in HPC cluster # HPC code -
HPCCall <- function(x,y){
summ <- data.frame("compound" = mydata$`Compound`[i])
allSumm <- addDataFrame(summ, sheet = ws_summary, col.names = TRUE)
}
#Cluster MQ Q Function
wb = createWorkbook()
ws_summary <- createSheet(wb, sheetName = "Summary")
clustermq::Q(HPCCall, x=1:nrow(mydata), const=list(y=mydata), export = list(wb=wb,ws_summary=ws_summary),pkgs= c("xlsx"), n_jobs = 2, log_worker=TRUE)
# Error
# jobs failed (0 warnings). Stopping.
# (Error #1) java.lang.NullPointerException |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Some objects can not be serialized, sent to another machine, and then unserialized. An example of this is objects that contain compiled code that accesses the memory of the local machine, as this will not be available on the remote. My recommendation would be not to serialize compiled code unless you know exactly what you're doing. Otherwise, you will run into memory issues like these. You can, however, send the content of the worksheet as |
Beta Was this translation helpful? Give feedback.
Some objects can not be serialized, sent to another machine, and then unserialized.
An example of this is objects that contain compiled code that accesses the memory of the local machine, as this will not be available on the remote.
My recommendation would be not to serialize compiled code unless you know exactly what you're doing. Otherwise, you will run into memory issues like these.
You can, however, send the content of the worksheet as
data.frames without problems!