Package: taskqueue 0.2.0

Bangyou Zheng

taskqueue: Task Queue for Parallel Computing Based on PostgreSQL

Implements a task queue system for asynchronous parallel computing using 'PostgreSQL' <https://www.postgresql.org/> as a backend. Designed for embarrassingly parallel problems where tasks do not communicate with each other. Dynamically distributes tasks to workers, handles uneven load balancing, and allows new workers to join at any time. Particularly useful for running large numbers of independent tasks on high-performance computing (HPC) clusters with 'SLURM' <https://slurm.schedmd.com/> job schedulers.

Authors:Bangyou Zheng [aut, cre]

taskqueue_0.2.0.tar.gz
taskqueue_0.2.0.zip(r-4.7-any)taskqueue_0.2.0.zip(r-4.6-any)taskqueue_0.2.0.zip(r-4.5-any)
taskqueue_0.2.0.tgz(r-4.6-any)taskqueue_0.2.0.tgz(r-4.5-any)
taskqueue_0.2.0.tar.gz(r-4.7-any)taskqueue_0.2.0.tar.gz(r-4.6-any)
taskqueue_0.2.0.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION
card.svg |card.png
taskqueue/json (API)

# Install 'taskqueue' in R:
install.packages('taskqueue', repos = c('https://byzheng.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/byzheng/taskqueue/issues

Pkgdown/docs site:https://taskqueue.bangyou.me

On CRAN:

Conda:

parallel-computingpostgresqltask-queue

5.08 score 2 stars 4 scripts 244 downloads 31 exports 61 dependencies

Last updated from:1859b3ae31. Checks:9 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-develOK141
source / vignettesOK202
linux-releaseOK159
macos-releaseOK110
macos-oldrelOK91
windows-develOK78
windows-releaseOK82
windows-oldrelOK75
wasm-releaseOK136

Exports:db_cleandb_connectdb_disconnectdb_initis_db_connectproject_addproject_deleteproject_getproject_listproject_resetproject_resource_addproject_resource_add_jobsproject_resource_getproject_resource_log_deleteproject_startproject_statusproject_stopresource_addresource_getresource_listshiny_apptask_addtask_cleantask_gettask_resettask_statustaskqueue_optionstaskqueue_resettq_applyworkerworker_slurm

Dependencies:askpassbase64encbitbit64blobbslibcachemclicommonmarkcpp11credentialscurlDBIdigestfarverfastmapfontawesomefsgenericsggplot2gluegtablehmshtmltoolshttpuvisobandjquerylibjsonlitelabelinglaterlifecyclelubridatemagrittrmemoisemimeopensslotelpkgconfigpromisesR6rappdirsRColorBrewerRcpprlangRPostgresS7sassscalessettingsshinysourcetoolssshstringistringrsystimechangevctrsviridisLitewhiskerwithrxtable

Getting Started with taskqueue
Introduction | Prerequisites | Installation | Basic Workflow | 1. Initialize Database | 2. Define a Resource | 3. Create a Project | 4. Assign Resource to Project | 5. Add Tasks | 6. Develop Your Worker Function | 7. Deploy to HPC | Task Status | Common Operations | Check Task Status | Reset Tasks | Manage Projects | Example: Complete Workflow | Tips
Rendered fromgetting-started.Rmdusingknitr::rmarkdown

Last update: 2025-11-24
Started: 2025-11-24

PostgreSQL Setup for taskqueue
Overview | Why PostgreSQL? | Installation on Ubuntu | Create Database and User | Configure Remote Access | 1. Edit postgresql.conf | 2. Edit pg_hba.conf | 3. Restart PostgreSQL | 4. Open Firewall if Needed | R Configuration | Install R Packages | Test Connection | Initialize taskqueue | Security Notes | Clean Database
Rendered frompostgresql-setup.Rmdusingknitr::rmarkdown

Last update: 2025-11-24
Started: 2025-11-24

SSH Setup for taskqueue
Overview | Prerequisites | Setup Steps | 1. Generate SSH keypair | 2. Copy the public key to the remote server | 3. Test the connection | 4. Test with R | Troubleshooting | Connection still asks for password | SSH key not found | Using SSH agent (optional, for keys with passphrases) | Security Best Practices | Additional Resources
Rendered fromssh-setup.Rmdusingknitr::rmarkdown

Last update: 2025-11-24
Started: 2025-11-24

Simple Workflow with tq_apply
Overview | Basic Usage | Function Arguments | Required Arguments | Optional Arguments | Passing Arguments to Your Function | Complete Example | Monitoring Progress | Collecting Results | Best Practices | 1. Save Results to Files | 2. Make Functions Idempotent | 3. Specify Working Directory | 4. Set Appropriate Resources | Comparison with Manual Workflow | Troubleshooting | When to Use tq_apply()
Rendered fromsimple-workflow.Rmdusingknitr::rmarkdown

Last update: 2025-11-24
Started: 2025-11-24

Readme and manuals

Help Manual

Help pageTopics
Check absolute path for system.check_absolute_path
Check absolute path for Linux.check_linux_absolute_path
Check absolute path for Windows.check_windows_absolute_path
Clean All Tables and Definitions from Databasedb_clean
Connect to PostgreSQL Databasedb_connect
Disconnect from PostgreSQL Databasedb_disconnect
Initialize PostgreSQL Database for taskqueuedb_init
A Wrapper function for DBI interfacedb_sql
Test Database Connectionis_db_connect
Create a New Projectproject_add
Delete a Projectproject_delete
Get Project Informationproject_get
List All Projectsproject_list
Reset a Projectproject_reset
Assign a Resource to a Projectproject_resource_add
Manage SLURM Job List for Project Resourceproject_resource_add_jobs
Get resources of a projectproject_resource_get
Delete Log Files for a Project Resourceproject_resource_log_delete
Start a Projectproject_start
Display Project Statusproject_status
Stop a Projectproject_stop
Add a New Computing Resourceresource_add
Get Information for a Specific Resourceresource_get
List All Computing Resourcesresource_list
Launch Shiny App for Monitoring Projectsshiny_app
Check whether a table is existedtable_exist
Add Tasks to a Projecttask_add
Remove All Tasks from a Projecttask_clean
Get Detailed Task Informationtask_get
Reset Task Status to Idletask_reset
Get Task Status Summarytask_status
Set or Get taskqueue Optionstaskqueue_options
Reset taskqueue Options to Defaultstaskqueue_reset
Apply a Function with Task Queue (Simplified Workflow)tq_apply
Execute Tasks as a Workerworker
Create a worker on slurm clusterworker_slurm