Sherlock changelog

A new tool to help optimize job resource requirements

2023-03-25T01:04:21.451Z

It’s not always easy to determine the right amount of resources to request for a computing job. Making sure that the application will have enough resources to run properly, but avoiding over-requests that would make the jobs spend too much time waiting in queue for resources they won’t be using.

To help users inform those choices, we’ve just added a new tool to the module list on Sherlock. ruse is command-line tool developed by Jan Moren which facilitates measuring processes’ resource usage. It periodically measures the resource use of a process and its sub-processes, and can help users find out how much resource to allocate to their jobs. It will determine the actual memory, execution time and cores that individual programs or MPI applications need to request in their job submission options.

You’ll find more information and some examples in the Sherlock documentation at https://www.sherlock.stanford.edu/docs/user-guide/running-jobs/#resource-requests

Hopefully ruse will make it easier to write job resource requests , and allow users to get a better understanding of their applications’ behavior to take better advantage of Sherlock’s capabilities.

As usual, if you have any question or comment, please don’t hesitate to reach out at [email protected]" rel="noopener" target="_blank">[email protected].

Job #1, again!

2022-11-06T02:11:25.989Z

This is not the first time, we’ve been through this already (not so long ago, actually) but today, the Slurm job id counter was reset and went from job #67043327 back to job #1.

JobID Partition Start
------------ ---------- -------------------
67043327 normal 2022-11-05T10:18:32
1 normal 2022-11-05T10:18:32

The largest job id that the scheduler can assign on Sherlock is 67,043,327. So when that number is reached, the next submitted job will be assigned job id #1.

This is the second time this job id reset happens in Sherlock’s history, since it debuted in 2014. The first occurrence happened on May 11th, 2020, just a little under 2.5 years ago.

It took about 6 years to submit the first 67 million jobs on Sherlock, but It’s incredible to realize that it took less than half that time to get to that staggering number of submitted jobs once again, and that it all happened since the beginning of the pandemic.

This is an humbling illustration of Sherlock’s central role and its importance to the Stanford research community, especially over the last few months. This give us once again the opportunity to thank each and every one of you, Sherlock users, for your continuous support, your extraordinary motivation and all of your patience and understanding when things break. We’ve never been so proud of supporting your amazing work, especially during those particularly trying times. Stay safe and happy computing!

A new interactive step in Slurm

2021-06-03T20:18:40.986Z

A new version of the sh_dev tool has been released, that leverages a recently-added Slurm feature.

Slurm 20.11 introduced a new “interactive step” , designed to be used with salloc to automatically launch a terminal on an allocated compute node. This new type of job step resolves a number of problems with the previous interactive job approaches, both in terms of accounting and resource allocation.

What is this about?

In previous versions, launching an interactive job with srun --pty bash would create a step 0, that was consuming resources, especially Generic Resources (GRES, ie. GPUs). Among other things, it made it impossible to use srun within that allocation to launch subsequent steps. Any attempt would result in a “step creation temporarily disabled” error message.

Now, with this new feature, you can use salloc to directly open a shell on a compute node. The new interactive step won’t consume any of the allocated resources, so you’ll be able to start additional steps with srun within your allocation.

sh_dev (aka sdev) has been updated to use interactive steps.

What changes?

For `sh_dev`

On the surface, nothing changes: you can continue to use sh_dev exactly like before, to start an interactive session on one of the compute nodes dedicated to that task (the default), or on a node in any partition (which is particularly popular among node owners). You’ll be able to use the same options, with the same features (including X11 forwarding).
Under the hood, though, you’ll be leveraging the new interactive step automatically.

For `salloc`

If you use salloc on a regular basis, the main change is that the resulting shell will open on the first allocated node, instead of the node you ran salloc on:

[kilian@sh01-ln01 login ~]$ salloc
salloc: job 25753490 has been allocated resources
salloc: Granted job allocation 25753490
salloc: Nodes sh02-01n46 are ready for job
[kilian@sh02-01n46 ~] (job 25753490) $

If you want to keep that initial shell on the submission host, you can simply specify a command as an argument, and the resulting command will continue to be executed as the calling user on the calling host:

[kilian@sh01-ln01 login ~]$ salloc bash
salloc: job 25752889 has been allocated resources
salloc: Granted job allocation 25752889
salloc: Nodes sh02-01n46 are ready for job
[kilian@sh01-ln01 login ~] (job 25752889) $

For `srun`

If you’re used to run srun —pty bash to get a shell on a compute node, you can continue to do so (as long as you don’t intend to run additional steps within the allocation).

But you can also just type salloc, get a more usable shell, and save 60% in keystrokes!

Happy computing! And as usual, please feel free to [email protected]" rel="noopener" target="_blank">reach out if you have comments or questions.

Job #1

2020-05-11T20:09:00.001Z

If you’ve been submitting jobs on Sherlock over the last couple days, you probably noticed something different about your your job ids… They lost a couple digits!

If you submitted a job last week, its job id was likely in the 67,000,000s. Today, it’s back in the 100,000s. What happened? Did we reset anything? Did we start simplifying job ids because there were too many numbers to keep track of?

Not really.

It’s just that so many jobs are submitted to Sherlock these days (and even more so since the beginning of the stay-at-home directives), that we reached the maximum job id that the scheduler can use.

Those job ids are roughly 26 bits in length, with a little headroom for special cases, and the largest job id that the scheduler can assign on Sherlock is 67,043,327. It means that when that number is reached, the next submitted job will be assigned jobid #1.

Both were submitted on Friday night, and started running Saturday morning:

 JobID Partition Submit Start
------------ ---------- ------------------- -------------------
1 normal 2020-05-08T22:21:28 2020-05-09T06:04:50
67043327 normal 2020-05-08T22:21:28 2020-05-09T06:05:16

A few months ago, we celebrated job #50,000,000. Today, we’re celebrating job #1, the beginning of a new cycle. :)

Thanks to each and every one of you, Sherlock users, for your continuous support, your extraordinary motivation and all of your patience and understanding when things break. We’ve never been so proud of supporting your amazing work, especially during those particularly trying times. Stay safe and happy computing!

More (and easier!) GPU scheduling options

2019-11-05T20:00:00.001Z

GPU scheduling is now easier and more powerful on Sherlock, with the addition of new job submission options especially targeted at GPU workloads.

The most visible change is that you can now use the --gpus option when submitting jobs, like this:

$ srun -p gpu --gpus=2 ...

A number of additional submission options can now be used, such as:

--cpus-per-gpu, to request a number of CPUs per allocated GPU,
--gpus-per-node, to request a given number of GPUs per node,
--gpus-per-task, to request a number of GPUs per spawned task,
--mem-per-gpu, to allocate a given amount of host memory per GPU.

You can now also allocate a different number of GPUs per node on multi-node jobs, change the frequency of the GPUs allocated to your job and explicitly set task-to-GPU binding maps.

All of those options are detailed in the updated documentations at https://www.sherlock.stanford.edu/docs/user-guide/gpu/ and a more complete description is available in the Slurm manual

Under the hood, the scheduler is now fully aware of the specifics of each GPU node, it knows how GPUs on the same node are inter-connected, and how they map to CPU sockets, and can select preferred GPUs for co-scheduling. It has all the information it needs to take optimal decisions about the placement of tasks within a job.

The end result? Better performance with less hassle for multi-GPU jobs.

So please take the new options for a spin, and [email protected]" target="_blank" rel="noopener">let us know how they work for your jobs!

A better view at Sherlock's resources

2019-05-03T22:36:00.001Z

How many jobs are running?
What partitions do I have access to?
How many CPUs can I use?
Where should I submit my jobs?

Any of those sound familiar?

We know it’s not always easy to navigate the native scheduler tools, their syntax, and the gazillion options they provide.

Enter `sh_part`

So today, we’re introducing sh_part^[1], a new command on Sherlock, that will simplify navigating Sherlock’s partitions, and provide an user-focused, centralized view of its computing resources.

To run it, simply type sh_part at the prompt on any login or compute node, and you’ll be greeted by something like this:

$ sh_part
QUEUE FREE TOTAL FREE TOTAL RESORC OTHER MAXJOBTIME CORES NODE GRES
PARTITION CORES CORES NODES NODES PENDNG PENDNG DAY-HR:MN PERNODE MEM-GB (COUNT)
normal* 30 1600 0 76 2801 2278 7-00:00 20-24 128-191 -
bigmem 0 88 0 2 90 1 1-00:00 32-56 512-3072 -
dev 50 56 2 3 32 0 0-02:00 16-20 128 -
gpu 62 140 0 7 121 0 7-00:00 16-24 191-256 gpu:8(1),gpu:4(6)

You’ll find a brief list of partitions you have access to, complete with information about the number of available nodes/cores and pending jobs.

in the QUEUE PARTITION column, the * character indicates the default partition.
the RESOURCE PENDING column shows the core count of pending jobs that are waiting on resources,
the OTHER PENDING column lists core counts for jobs that are pending for other reasons, such as licenses, user, group or any other limit,
the GRES column shows the number and type of GRES available in that partition, and the number of nodes that feature that specific GRES combination in paranteses. So for instance, in the output above, the gpu partition features ` node with 8 GPUs, and 6 nodes with 4 GPUs each.

Hopefully sh_part will make it easier to figure out cluster activity, and allow users to get a better understanding of what’s running and what’s available in the various Sherlock partitions.

As usual, if you have any question or comment, please don’t hesitate to reach out at [email protected]" target="_blank" rel="noopener">[email protected].

sh_part is based on the spart tool, written by Ahmet Mercan. ↩

Persistent processes on Sherlock

2018-11-05T22:36:00.001Z

There are many cases where having a persistent process (or service) running alongside computing jobs is of great benefit. For instance, when data is stored in a database format to allow for highly efficient queries, jobs that want to compute against this data will need to retrieve it from a live database instance.

We’re providing instructions and examples in the Sherlock documentation on how to run database server instances in the context of a job (with MariaDB and PostgreSQL), that is, subject to the regular execution time limits of a scheduled job. But what if you need those instances to run all the time?

Now, with persistent jobs, users can submit jobs on Sherlock that will resubmit themselves when they reach their time limit, and can also conserve their $JOBID across re-submissions. This will make specifying job dependencies much easier, since the persistent job’s id will never change.

For more details, and examples, please take a look at the persistent jobs documentation and [email protected]" target="_blank">let us know if you have any question.

Better error messages when submitting jobs

2018-09-18T21:53:00.001Z

Sherlock now offers a better and more complete explanation when a job submission is rejected by the scheduler.

What does it look like?

In the most common cases, jobs that don’t meet the requirements for the partition they’re submitted to, will display a more detailed message.

For instance, submitting a job to the gpu partition without requesting a GPU will look like this:

$ srun -p gpu --pty bash
srun: error:
=============================================================================
ERROR: missing GPU request, job not submitted
=============================================================================
Jobs submitted to the gpu partition must explicitly request GPUs, by using
the --gres option.
-----------------------------------------------------------------------------
srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

We hope this will make things easier when a job is rejected at submission time, and help clarify some of the errors sent back by the scheduler.

Don’t hesitate to [email protected]" target="_blank">contact us if you have any feedback or suggestion.

High priority QOS for owners

2018-08-01T23:32:00.001Z

Today, we're introducing a new high-priority QOS for owner partitions.

In groups which have purchased their own compute nodes on Sherlock, users can now submit jobs to their group partition using the --qos=high_p option: it will give those jobs a priority boost, and will allow for distinguishing between two classes of jobs within the same partition.

This new QOS could be useful to prioritize specific jobs that need to get executed before others, to meet deadlines or differentiate between background and higher-priority jobs.

If your group is not an owner on Sherlock yet, but if you're interested in becoming one, please take a look at the Sherlock website and [email protected]" target="_blank">let us know if you have any question.