Sherlock changelog

Instant lightweight GPU instances are now available

2023-04-27T01:05:18.100Z

We know that getting access to GPUs on Sherlock can be difficult and feel a little frustrating at times. Demand has been steadily growing, leading to long pending times, and waiting in line rarely feels great, especially when you have important work to do.

Which is why we are excited to announce the immediate availability of our latest addition to the Sherlock cluster: instant lightweight GPU instances! Every user can now get immediate access to a GPU instance, for a quick debugging session or to explore new ideas in a Notebook.

GPUs are the backbone of high-performance computing. They’ve become an integral component of the toolbox for many users, and are essential for deep learning, scientific simulations, and many other applications. But you don’t always need a full-fledged, top-of-the-line GPU for all your tasks. Sometimes all you want is to run a quick test to prototype an idea, debug a script, or explore new data in an interactive Notebook. For this, the new lightweight GPU instances on Sherlock will give you instant access to a GPU, without having to wait in line and compete with other jobs for resources you don’t need.

Sherlock’s instant lightweight GPU instances leverage NVIDIA’s Multi-Instance GPU (MIG) to provide multiple fully isolated GPU instances on the same physical GPU, each with their own high-bandwidth memory, cache, and compute cores. Those lightweight instances are ideal for small to medium-sized jobs, and lower the barrier to entry for all users

Similar to the interactive sessions available through the dev partition, Sherlock users can now request a GPU via the sh_dev command, and get immediate access with the following command:

$ sh_dev -g 1

For interactive apps in the Sherlock OnDemand interface, requesting a GPU in the dev partition will initiate an interactive session with access to a lightweight GPU instance.

So now, everyone gets a GPU, no questions asked! 😁

We hope those new instances will improve access to GPUs on Sherlock, enable a wider range of use cases, with all the flexibility and performance you need to get your work done, and lead to even more groundbreaking discoveries!

As always, thanks to all of our users for your continuous support and patience as we work to improve Sherlock, and if you have any question or comment, please don’t hesitate to reach out at [email protected]" rel="noopener" target="_blank">[email protected].

ClusterShell on Sherlock

2022-12-03T02:57:22.756Z

Ever wondered how your jobs were doing while they were running? Keeping a eye on a log file is nice, but what if you could quickly gather process lists, usage metrics and other data points from all the nodes your multi-node jobs are running on, all at once?

Enter ClusterShell, the best parallel shell application (and library!) of its kind.

With ClusterShell on Sherlock, you can quickly run a command on all the nodes your job is running on, to gather information about your applications and processes, in real time, and gather live output without having to wait for your job to end to see how it did. And with its tight integration with the job scheduler, no need to fiddle with manual node lists anymore, all it needs is a job id!

You allocated a few nodes in an interactive session and want to distribute some files on each node’s local storage devices? Check: ClusterShell has a copy mode just for this.

Want to double-check that your processes are correctly laid out? Check: you can run a quick command to check the process tree across the nodes allocated to your job with:

$ clush -w @job:$JOBID pstree -au $USER

and verify that all your processes are running correctly.

You’ll find more details and examples in our Sherlock documentation, at https://www.sherlock.stanford.edu/docs/software/using/clustershell

Questions, ideas, or suggestions? Don’t hesitate to reach out to [email protected]" rel="noopener nofollow" target="_blank">[email protected] to let us know!

New GPU options in the Sherlock catalog

2020-09-18T18:00:00.001Z

Today, we’re introducing the latest generation of GPU accelerators in the Sherlock catalog: the NVIDIA A100 Tensor Core GPU.

Each A100 GPU features 9.7 TFlops of double-precision (FP64) performance, up to 312 TFlops for deep-learning applications, 40GB of HBM2 memory, and 600GB/s of interconnect bandwidth with 3rd generation NVLink connections^[1].

New Sherlock Catalog options

Targeting the most demanding HPC and DL/AI workloads, the three new GPU node options we’re introducing today should cover the most extreme computing needs:

a refreshed version of the SH3_G4FP64.1 configuration features 32x CPU cores, 256GB of memory and 4x A100 PCIe GPUs
the new SH3_G4TF64 model features 64 CPU cores, 512GB of RAM, and 4x A100 SXM4 GPUs (NVLink)
and the most powerful configuration, SH3_G8TF64 , comes with 128 CPU cores, 1TB of RAM, 8x A100 SXM4 GPUs (NVLink) and two Infiniband HDR HCAs for a whopping 400Gb/s of interconnect bandwidth to keep those GPUs busy

You’ll find all the details in the Sherlock catalog (SUNet ID required).

All those configuration are available for order today, and can be ordered online though the Sherlock order form (SUNet ID required).

Other models’ availability

We’re working on bringing a replacement for the entry-level SH3_G4FP32 model back in the catalog as soon as possible. We’re unfortunately dependent on GPU availability, as well as on the adaptations required for server vendors to accommodate the latest generation of consumer-grade GPUs. We’re expecting a replacement configuration in the same price range to be available by the end of the calendar year.

As usual, please don’t hesitate to [email protected]" target="_blank" rel="noopener">reach out if you have any questions!

In-depth technical details are available in the NVIDIA Developer blog ↩

New Sherlock on-boarding sessions

2020-05-18T17:23:00.001Z

One of the most requested improvements around Sherlock services, that came out of our recent user survey, was for more documentation and more training.

This is why, to help new users get familiar with Sherlock’s computing environment, we’ll now be offering regular Sherlock on-boarding sessions, starting this Thursday, via Zoom:

Sherlock on-boarding session
Thursday, May 21st, 11am
https://stanford.zoom.us/j/93659504122
SUNetID required

This one-hour introduction to Sherlock will present the cluster’s layout, the job scheduler and its limits, the different data storage possibilities, as well as some job submission and software installation best practices. So if you’re new to Sherlock or HPC in general, you’re welcome to join us to learn more!

We will be offering these sessions on a regular basis, so look out for more announcements soon.

On-boarding sessions will be focusing on new Sherlock users, but if you’re already a Sherlock user and have specific questions, always feel free to reach out to us at [email protected]" target="_blank" rel="noopener">[email protected], or to stop by during virtual office hours:

Tuesdays, 10-11am: https://stanford.zoom.us/j/901884213
Thursdays, 3-4pm: https://stanford.zoom.us/j/681964418

A newer, faster and better /scratch

2019-12-03T23:30:00.001Z

As we just announced, Sherlock now features a brand new storage system for /scratch. But what was the old system, what does the new one look like, and how did the move happen? Read on to find out!

The old

Since its early days, Sherlock ran its /scratch filesystem on a storage system that was donated by Intel and Dell.

Dubbed Regal, it was one of the key components of the Sherlock cluster when we started it in early 2014, with an initial footprint of about 100 compute nodes. Its very existence allowed us to scale the cluster to more than 1,500 nodes today, almost entirely through Faculty and PIs contributions to its condominium model. That’s a 15x growth in 5 years, and adoption has been spectacular.

Regal was initially just over 1PB when it’s been deployed in May 2013, which was quite substantial at the time. And similarly to the compute part of the cluster, its modular design allowed us to expand it to over 3PB with contributions from individual research groups.

We had a number of adventures with that system, including a major scale disk replacement operation, where we replaced about a petabyte of hard drives in production, while continuing to serve files to users ; or a literal drawer explosion in one of the disk arrays!

It’s been fun, and again, invaluable to our users.

But time has come to retire it, and replace it with a newer, faster and better solution, to accommodate the ever-growing storage needs of Sherlock’s ever-growing community.

The new

This year, we stood up a completely new and separate /scratch filesystem for Sherlock.

Nicknamed Fir (we like trees), this new storage system features:

multiple metadata servers and faster metadata storage for better responsiveness with interactive operations,
faster object storage servers,
a faster backend interconnect, for lower latency operations across storage servers,
more and faster storage routers to provide more bandwidth from Sherlock to /scratch,
more space to share amongst all Sherlock users,
a newer version of Lustre which provides:
- improved client performance,
- dynamic file striping to automatically adapt file layout and I/O performance to match a file’s size
- and much more!

And not to brag, but Fir has been ranked #15 in the IO-500 list of the fastest storage systems in the world, in the 10-node challenge category, that was released at SC’19. So yes, it’s decently fast.

The migration

Now, usually, when a new filesystem is made available on a computing system, there are two approaches:

One is making the new system available under a new mount point (like /scratch2) and tell users: “here’s the new filesystem, the old one will go away soon, you have until next Monday to get your files there and update all your scripts.”
This usually results in a lot of I/O traffic going on at once from all the users rushing to copy their data to the new space, potential mistakes, confusion, and in the end, a lot of frustration, additional work and unnecessary stress on everyone. Not good.

The other one is for sysadmins to copy all of the existing data from the old system to the new one in the background, in several passes, and then scheduled a (usually long) downtime to run a last synchronization pass and substitute the old filesystem by the new one under the same mount point (/scratch).
This also brings significant load on the filesystem while the synchronisation passes are running, taking I/O resources away from legitimate user jobs, it’s usually a very long process, and in the end it brings over old and abandoned files to the new storage system, wasting precious space. Not optimal either.

So we decided to take another route, and devised a new scheme. We spent some time (and fun!) designing and developing a new kind of overlay layer, to bridge the gap between Regal and Fir, and to transparently migrate user data from one to the other.

We (aptly) named this layer migratefs and open-sourced it at:
https://github.com/stanford-rc/fuse-migratefs.

The main idea of migratefs is to take advantage of user activity to:

distribute the data transfer tasks across all of the cluster nodes, to reduce the overall migration time,
only migrate data that is actively in use, and leave older files that are never accessed nor modified on the old storage system, resulting in a new storage system that only stores relevant data,
migrate all the user data transparently, without any downtime.

So over the last few months, all of the active user data on Regal has been seamlessly migrated to Fir, without users having to modify any of their job scripts, and all without a downtime.

Which is why if you’re using $SCRATCH or $GROUP_SCRATCH today, you are actively using the new storage system, and all your active data is there already, ready to be used in your compute jobs.

Next steps

Now, Regal has been emptied of all of its data and has been retired. It’s currently being un-racked to make room for future Sherlock developments.And stay tuned, because… epic changes are coming!

More (and easier!) GPU scheduling options

2019-11-05T20:00:00.001Z

GPU scheduling is now easier and more powerful on Sherlock, with the addition of new job submission options especially targeted at GPU workloads.

The most visible change is that you can now use the --gpus option when submitting jobs, like this:

$ srun -p gpu --gpus=2 ...

A number of additional submission options can now be used, such as:

--cpus-per-gpu, to request a number of CPUs per allocated GPU,
--gpus-per-node, to request a given number of GPUs per node,
--gpus-per-task, to request a number of GPUs per spawned task,
--mem-per-gpu, to allocate a given amount of host memory per GPU.

You can now also allocate a different number of GPUs per node on multi-node jobs, change the frequency of the GPUs allocated to your job and explicitly set task-to-GPU binding maps.

All of those options are detailed in the updated documentations at https://www.sherlock.stanford.edu/docs/user-guide/gpu/ and a more complete description is available in the Slurm manual

Under the hood, the scheduler is now fully aware of the specifics of each GPU node, it knows how GPUs on the same node are inter-connected, and how they map to CPU sockets, and can select preferred GPUs for co-scheduling. It has all the information it needs to take optimal decisions about the placement of tasks within a job.

The end result? Better performance with less hassle for multi-GPU jobs.

So please take the new options for a spin, and [email protected]" target="_blank" rel="noopener">let us know how they work for your jobs!

A better view at Sherlock's resources

2019-05-03T22:36:00.001Z

How many jobs are running?
What partitions do I have access to?
How many CPUs can I use?
Where should I submit my jobs?

Any of those sound familiar?

We know it’s not always easy to navigate the native scheduler tools, their syntax, and the gazillion options they provide.

Enter `sh_part`

So today, we’re introducing sh_part^[1], a new command on Sherlock, that will simplify navigating Sherlock’s partitions, and provide an user-focused, centralized view of its computing resources.

To run it, simply type sh_part at the prompt on any login or compute node, and you’ll be greeted by something like this:

$ sh_part
QUEUE FREE TOTAL FREE TOTAL RESORC OTHER MAXJOBTIME CORES NODE GRES
PARTITION CORES CORES NODES NODES PENDNG PENDNG DAY-HR:MN PERNODE MEM-GB (COUNT)
normal* 30 1600 0 76 2801 2278 7-00:00 20-24 128-191 -
bigmem 0 88 0 2 90 1 1-00:00 32-56 512-3072 -
dev 50 56 2 3 32 0 0-02:00 16-20 128 -
gpu 62 140 0 7 121 0 7-00:00 16-24 191-256 gpu:8(1),gpu:4(6)

You’ll find a brief list of partitions you have access to, complete with information about the number of available nodes/cores and pending jobs.

in the QUEUE PARTITION column, the * character indicates the default partition.
the RESOURCE PENDING column shows the core count of pending jobs that are waiting on resources,
the OTHER PENDING column lists core counts for jobs that are pending for other reasons, such as licenses, user, group or any other limit,
the GRES column shows the number and type of GRES available in that partition, and the number of nodes that feature that specific GRES combination in paranteses. So for instance, in the output above, the gpu partition features ` node with 8 GPUs, and 6 nodes with 4 GPUs each.

Hopefully sh_part will make it easier to figure out cluster activity, and allow users to get a better understanding of what’s running and what’s available in the various Sherlock partitions.

As usual, if you have any question or comment, please don’t hesitate to reach out at [email protected]" target="_blank" rel="noopener">[email protected].

sh_part is based on the spart tool, written by Ahmet Mercan. ↩

New GPU node available on Sherlock

2019-02-16T00:35:00.001Z

There’s a new GPU node in the gpu partition!

It’s notable for a list of reasons. This is the first node on Sherlock to feature both:

the latest generation of Intel CPUs (Skylake),
the latest generation of computing-optimized NVIDIA GPUs,

and it’s also the first node on Sherlock with 32GB GPUs, which is particularly interesting for a lot of Deep-Learning oriented workloads.

Specs

This compute node features:

24x Intel Skylake CPU cores (2x Xeon 5118, 2.30GHz),
192GB of memory (RAM),
4x NVIDIA Tesla V100 GPUs with 32GB of GPU memory each.

Details

Nodes in the gpu partition are now available to everyone on Sherlock, and the new node can be requested by adding the following flag to your job submission options: -C "GPU_SKU:V100_SXM2&GPU_MEM:32GB"

To request an interactive session on a Tesla V100 GPU with 32GB of memory, you can run:

$ srun -p gpu --gres gpu:1 -C "GPU_SKU:V100_SXM2&GPU_MEM:32GB" --pty bash

To see the list of all the available GPU features and characteristics that can be requested in the gpu partition:

$ sh_node_feat -p gpu | grep GPU
GPU_BRD:GEFORCE
GPU_BRD:TESLA
GPU_CC:3.5
GPU_CC:3.7
GPU_CC:5.2
GPU_CC:6.0
GPU_CC:6.1
GPU_CC:7.0
GPU_GEN:KPL
GPU_GEN:MXW
GPU_GEN:PSC
GPU_GEN:VLT
GPU_MEM:12GB
GPU_MEM:16GB
GPU_MEM:24GB
GPU_MEM:32GB
GPU_MEM:6GB
GPU_SKU:K20X
GPU_SKU:K80
GPU_SKU:P100_PCIE
GPU_SKU:P100_SXM2
GPU_SKU:P40
GPU_SKU:TITAN_BLACK
GPU_SKU:TITAN_X
GPU_SKU:TITAN_Xp
GPU_SKU:V100_SXM2

For more details about GPUs on Sherlock, see the GPU user guide.

If you have any question, feel free to send us a note at [email protected]" target="_blank" rel="noopener">[email protected].

A better way to check quotas on Sherlock

2019-02-14T23:05:00.001Z

We’re very pleased to introduce a new way to check data usage on Sherlock, all from a single command, and using an hopefully simpler way to display information than before.

Introducing `sh_quota`

sh_quota, the new quota checking tool for Sherlock, displays quota usage on the different Sherlock filesystems, using a familiar and consistent format:

It can be used to display usage on a single filesystem, or in the context of a different group, for users who are affiliated to multiple groups on Sherlock.

You probably also noticed that quota information is now automatically displayed when you login onto Sherlock. This provides a quick and easy way to see if any of the limits is reached, and save some time in diagnosing errors later on, if your $HOME quota is exceeded, for instance.

If you don’t want your quota information to be displayed at login, you can easily disable them by creating a ~/.sh_noquota file in your $HOME directory:

$ touch ~/.sh_noquota

and the status information will be gone the next time you connect.

For complete details and usage examples, please refer to the Checking Quotas section of the Sherlock storage documentation.

Sherlock OnDemand

2018-11-22T01:00:00.001Z

Today, we’re announcing Sherlock OnDemand, a brand new way to use the Sherlock cluster.

Hot on the heels of the SC18 Supercomputing Conference, and right in time for the long Thanksgiving week-end, we thought that a good way to thank our users, who, from grad students to Faculty members, have been showing their appreciation and unfaltering support over the years, would be to demonstrate our commitment to provide them with innovative and easier ways to use computing resources to support their research.

This is why, after a long wait, we’re extremely pleased to announce the immediate availability of the new Sherlock OnDemand service.

A revolutionary way to work on Sherlock

Sherlock OnDemand is a completely new way to interact with the computing and data storage resources provided on Sherlock.

From the comfort of their web browser, users can now connect to Sherlock, compose, submit and monitor jobs, manage their files, and run interactive applications, such as Jupyter Notebooks, RStudio, or Tensorboard sessions.

A Sherlock shell from your browser

Yes, that means that you can now connect to Sherlock from your web browser. No SSH client required. No need to mess around in /.ssh/config anymore, no more Kerberos headaches, no more repeated two-step authentication confirmation either. Windows users rejoice!

Just point your browser to the Sherlock OnDemand login URL and shell away!

So long WinSCP!

Ever dreamed about being able to browse your files on Sherlock in a graphical way, without having to install additional programs, or mounting Sherlock’s filesystems on your local machine (sometimes awkwardly, if possible at all)?

Ever been frustrated that the Globus web interface didn’t offer a "right click > download" option?

Well, not only does Sherlock OnDemand allow all of this without the blink of an eye, but you can now view, edit, manipulate and transfer your files to (and from) Sherlock, from the comfort of your regular web browser.

All your jobs belong to Sherlock OnDemand.

Check out the queue, submit new jobs, cancel the ones you don’t like. All in one place, not a single shell command involved. Check this out!

Interactive apps

And🍒on top, interactive apps!

You can now start Jupyter Notebooks or Rstudio directly from your web browser. No more SSH tunnel to configure or convoluted setup process. Choose the application of your choice in the list of available apps, fill out the form to tune your session to your computing needs, and submit. Your interactive app will be scheduled on a compute node, and you’ll be able to connect to it from the click of a button.

Jupyter, RStudio, and Tensorboard are just the beginning! Stay tuned for more apps, coming soon to an Sherlock OnDemand browser tab near you.

Documentation

For complete details about Sherlock OnDemand, please see the documentation we’ve prepared.

And as usual, if you have any question, comment or suggestion, don’t hesitate to reach out at [email protected]" target="_blank">[email protected].

Happy computing!