tpar

simple, parallel job scheduling.

tpar is a simple job scheduling and dispatch service for distributing and monitoring tasks across machines. It was written to serve as a simple and easy-to-administer substitute for systems like Grid Engine.

Configuring a tpar instance is simply a matter of running tpar server on a designated server machine,

$ # We'll need to know the hostname of the server
$ hostname
my-server
$ # Start a server also running 8 local workers
$ tpar server -Hmy-server -N8

Submitting a job is then similarly easy,

$ tpar enqueue -H hostname -- long-process arg1 arg2

One can then enqueue jobs easily

$ tpar enqueue -Hmy-server -- bash -c "primes 10000000000  | head -n1"
$ tpar status -v
0     unnamed-job                                        finished
priority:       0
queued:         1 seconds ago
command:        bash
arguments:      -c primes 10000000000  | head -n1
logging:        stdout: none
stderr: none
status:         finished with 0 (at 1 seconds ago)
started at 1 seconds ago
ran on nid://localhost:5757:0

One can add more workers to help churn through the work queue using the tpar worker command,

$ # Add 16 more workers running on another machine
$ ssh my-workers -- tpar worker -Hmy-server -N16

Finally, the output of running jobs can be monitored using the tpar watch command,

$ tpar watch id=0