我想知道如何使用R函数在Rscript中运行500个并行作业。我目前有一个R文件，顶部有标题：
args <- commandArgs(TRUE)
B <- as.numeric(args[1])
Num.Cores <- as.numeric(args[2])
在R文件之外，我希望传递500个作业中的哪一个，这是由B指定的。另外，我想控制每个作业可用的核心/ CPU数量，Num.Cores。
我想知道是否有软件或指南可以允许这个。我目前有一个CentOS 7 / Linux服务器，我知道一种方法是安装Slurm。然而，这是一个非常麻烦的事情，我想知道是否有办法执行500个带有队列的作业。谢谢。
这就是我使用SLURM调度程序在集群上设置的方法
slurm sbatch工作提交脚本
#!/bin/bash
#SBATCH --partition=xxx ### Partition (like a queue in PBS)
#SBATCH --job-name=array_example ### Job Name
#SBATCH -o jarray.%j.%N.out ### File in which to store job output/error
#SBATCH --time=00-00:30:00 ### Wall clock time limit in Days-HH:MM:SS
#SBATCH --nodes=1 ### Node count required for the job
#SBATCH --ntasks=1 ### Nuber of tasks to be launched per Node
#SBATCH --cpus-per-task=2 ### Number of threads per task (OMP threads)
#SBATCH --mail-type=FAIL ### When to send mail
#SBATCH [email protected]
#SBATCH --get-user-env ### Import your user environment setup
#SBATCH --requeue ### On failure, requeue for another try
#SBATCH --verbose ### Increase informational messages
#SBATCH --array=1-500%50 ### Array index | %50: number of simultaneously tasks
echo
echo "****************************************************************************"
echo "* *"
echo "********************** sbatch script for array job *************************"
echo "* *"
echo "****************************************************************************"
echo
current_dir=${PWD##*/}
echo "Current dir: $current_dir"
echo
pwd
echo
# First we ensure a clean running environment:
module purge
# Load R
module load R/R-3.5.0
### Initialization
# Get Array ID
i=${SLURM_ARRAY_TASK_ID}
# Output file
outFile="output_parameter_${i}.txt"
# Pass line #i to a R script
Rscript --vanilla my_R_script.R ${i} ${outFile}
echo
echo '******************** FINISHED ***********************'
echo
my_R_script.R从arg剧本中获取sbatch
args <- commandArgs(trailingOnly = TRUE)
str(args)
cat(args, sep = "\n")
# test if there is at least one argument: if not, return an error
if (length(args) == 0) {
stop("At least one argument must be supplied (input file).\n", call. = FALSE)
} else if (length(args) == 1) {
# default output file
args[2] = "out.txt"
}
cat("\n")
print("Hello World !!!")
cat("\n")
print(paste0("i = ", as.numeric(args[1])))
print(paste0("outFile = ", args[2]))
### Parallel:
# https://hpc.nih.gov/apps/R.html
# https://github.com/tobigithub/R-parallel/blob/gh-pages/R/code-setups/Install-doSNOW-parallel-DeLuxe.R
# load doSnow and (parallel for CPU info) library
library(doSNOW)
library(parallel)
detectBatchCPUs <- function() {
ncores <- as.integer(Sys.getenv("SLURM_CPUS_PER_TASK"))
if (is.na(ncores)) {
ncores <- as.integer(Sys.getenv("SLURM_JOB_CPUS_PER_NODE"))
}
if (is.na(ncores)) {
return(2) # default
}
return(ncores)
}
ncpus <- detectBatchCPUs()
# or ncpus <- future::availableCores()
cat(ncpus, " cores detected.")
cluster = makeCluster(ncpus)
# register the cluster
registerDoSNOW(cluster)
# get info
getDoParWorkers(); getDoParName();
##### insert parallel computation here #####
# stop cluster and remove clients
stopCluster(cluster); print("Cluster stopped.")
# insert serial backend, otherwise error in repetitive tasks
registerDoSEQ()
# clean up a bit.
invisible(gc); remove(ncpus); remove(cluster);
# END
P.S：如果你想逐行读取参数文件，请在sbatch脚本中包含以下行，然后将它们传递给my_R_script.R
### Parameter file to read
parameter_file="parameter_file.txt"
echo "Parameter file: ${parameter_file}"
echo
# Read line #i from the parameter file
PARAMETERS=$(sed "${i}q;d" ${parameter_file})
echo "Parameters are: ${PARAMETERS}"
echo
参考文献：