There are many cases where having a persistent process (or service) running alongside computing jobs is of great benefit. For instance, when data is stored in a database format to allow for highly efficient queries, jobs that want to compute against this data will need to retrieve it from a live database instance.

We’re providing instructions and examples in the Sherlock documentation on how to run database server instances in the context of a job (with MariaDB and PostgreSQL), that is, subject to the regular execution time limits of a scheduled job. But what if you need those instances to run all the time?

Now, with persistent jobs, users can submit jobs on Sherlock that will resubmit themselves when they reach their time limit, and can also conserve their $JOBID across re-submissions. This will make specifying job dependencies much easier, since the persistent job’s id will never change.

For more details, and examples, please take a look at the persistent jobs documentation and let us know if you have any question.