Sherlock changelog

Sherlock goes full flash

2024-02-07T23:49:24.699Z

What could be more frustrating than anxiously waiting for your computing job to finish? Slow I/O that makes it take even longer is certainly high on the list. But not anymore! Fir, Sherlock’s scratch file system, has just undergone a major tech face-lift: it’s now a 10 PB all-flash storage system, providing an aggregate bandwidth of 400 GB/sec (and >800 kIOPS). Bringing Sherlock’s high-performance parallel scratch file system into the era of flash storage was not just a routine maintenance task, but a significant leap into the future of HPC and AI computing.

But first, a little bit of context

Traditionally, High-Performance Computing clusters face a challenge when dealing with modern, data-intensive applications. Existing HPC storage systems, long designed with spinning disks to provide efficient and parallel sequential read/write operations, often become bottlenecks for modern workloads generated by AI/ML or CryoEM applications. Those demand substantial data storage and processing capabilities, putting a strain on traditional systems.

So to accommodate those new needs and future evolution of the HPC I/O landscape, we at Stanford Research Computing, with the generous support of the Vice Provost and Dean of Research, have been hard at work for over two years, revamping Sherlock's scratch with an all-flash system.

And it was not just a matter of taking delivery of a new turn-key system. As most things we do, it was done entirely in-house: from the original vendor-agnostic design, upgrade plan, budget requests, procurement, gradual in-place hardware replacement at the Stanford Research Computing Facility (SRCF), deployment and validation, performance benchmarks, to the final production stages, all of those steps were performed with minimum disruption for all Sherlock users.

The technical details

The /scratch file system on Sherlock is using Lustre, an open-source, parallel file system that supports many requirements of leadership class HPC environments. And as you probably know by now, Stanford Research Computing loves open source! We actively contribute to the Lustre community and are a proud member of OpenSFS, a non-profit industry organization that supports vendor-neutral development and promotion of Lustre.

In Lustre, file metadata and data are stored separately, with Object Storage Servers (OSS) serving file data on the network. Each OSS pair and associated storage devices forms an I/O cell, and Sherlock's scratch has just bid farewell to its old HDD-based I/O cells. In their place, new flash-based I/O cells have taken the stage, each equipped with 96 x 15.35TB SSDs, delivering mind-blowing performance.

Sherlock’s /scratch has 8 I/O cells and the goal was to replace every one of them. Our new I/O cell has 2 OSS with Infiniband HDR at 200Gb/s (or 25GB/s) connected to 4 storage chassis, each with 24 x 15.35TB SSD (dual-attached 12Gb/s SAS), as pictured below:

Of course, you can’t just replace each individual rotating hard-drive with a SSD, there are some infrastructure changes required, and some reconfiguration needed. The upgrade, executed between January 2023 and January 2024, was a seamless transition. Old HDD-based I/O cells were gracefully retired, one by one, while flash-based ones progressively replaced them, gradually boosting performance for all Sherlock users throughout the year.

All of those replacements happened while the file system was up and running, serving data to the thousands of computing jobs that run on Sherlock every day. Driven by our commitment to minimize disruptions to users, our top priority was to ensure uninterrupted access to data throughout the upgrade. Data migration is never fun, and we wanted to avoid having to ask users to manually transfer their files to a new, separate storage system. This is why we developed and contributed a new feature in Lustre, which allowed us to seamlessly remove existing storage devices from the file system, before the new flash drives could be added. More technical details about the upgrade have been presented during the LAD’22 conference.

Today, we are happy to announce that the upgrade is officially complete, and Sherlock stands proud with a whopping 9,824 TB of solid-state storage in production. No more spinning disks in sight!

Key benefits

For users, the immediately visible benefits are quicker access to their files, faster data transfers, shorter job execution times for I/O intensive applications. More specifically, every key metric has been improved:

IOPS: over 100x (results may vary, see below)
Backend bandwidth: 6x (128 GB/s to 768 GB/s)
Frontend bandwidth: 2x (200 GB/s to 400 GB/s)
Usable volume: 1.6x (6.1 PB to 9.8 PB)

In terms of measured improvement, the graph below shows the impact of moving to full-flash storage for reading data from 1, 8 and 16 compute nodes, compared to the previous /scratch file system:

And we even tried to replicate the I/O patterns of AlphaFold, a well-known AI model to predict protein structure, and the benefits are quite significant, with up to 125x speedups in some cases:

This upgrade is a major improvement that will benefit all Sherlock users, and Sherlock’s enhanced I/O capabilities will allow them to approach data-intensive tasks with unprecedented efficiency. We hope it will help support the ever-increasing computing needs of the Stanford research community, and enable even more breakthroughs and discoveries.

As usual, if you have any question or comment, please don’t hesitate to reach out to Research Computing at [email protected]" rel="noopener nofollow" target="_blank" title="[email protected]">[email protected]. 🚀🔧

A brand new Sherlock OnDemand experience

2023-11-16T02:00:00Z

Following a long tradition of announces and releases during the SuperComputing conference, and while SC23 is underway in Denver CO, Stanford Research Computing is proud to unveil Sherlock OnDemand 3.0, a cutting-edge enhancement to its computing and data storage resources, revolutionizing user interaction and efficiency.

The upgraded Sherlock OnDemand is available immediately at https://ondemand.sherlock.stanford.edu

This new release brings a host of transformative changes. A lot happened under the hood, but the visible changes are significant as well.

Infrastructure upgrades:

A new URL: Sherlock OnDemand is now accessible at https://ondemand.sherlock.stanford.edu, in line with our other instances, for a more homogeneous user experience across Research Computing systems. The previous URL will still work for a time, and redirections will be progressively deployed to ease the transition.
New engine, same feel: a lot of internal components have undergone substantial updates, but the familiar interface remains intact, ensuring a seamless transition for existing users.
Streamlined authentication: Sherlock OnDemand now uses OIDC via the Stanford central Identity Provider instead of SAML, resulting in a lighter, more robust configuration for enhanced security.
Enhanced Performance: expect a more responsive interface and improved reliability with the eradication of 422 HTTP errors.

User-centric features:

Expanded file access: all your Oak groups, are now conveniently listed in the embedded file browser for easier and more comprehensive access to your data. And if you have rclone remotes already configured on Sherlock, you’'ll find them there as well!
Effortless support tickets: you can now send support tickets directly from the OnDemand interface, which will automatically include contextual information about your interactive sessions, to simply issue resolution.
New interactive apps: In addition to the existing apps, VS Code server, MATLAB, and JupyterLab join the platform, offering expanded functionalities, like the ability of loading and unloading of modules within JupyterLab directly.
Yes, you read that right: we now have VS Code and MATLAB in Sherlock OnDemand!
The RStudio app has also been rebuilt from the ground up, providing a much better and reliable experience.
Customizable working directories: users can now select a working directory across all interactive apps, for easier customization of their work environment.

For more details and guidance on using the new features, check out the updated documentation at https://www.sherlock.stanford.edu/docs/user-guide/ondemand/.

This update delivers a brand new computing experience, designed to empower you in your work. Sherlock OnDemand 3.0 marks a significant milestone in optimizing user access to computing resources, lowering the barrier to entry for new users, and empowering researchers with an unparalleled computing environment. We're excited to see how it will enhance your productivity and efficiency, so dive into this transformative experience today and elevate your computing endeavors to new heights with Sherlock OnDemand 3.0!

And as usual, if you have any question, comment or suggestion, don’t hesitate to reach out at [email protected]" rel="noopener nofollow" target="_blank" title="support">[email protected].

Final hours announced for the June 2023 SRCF downtime

2023-05-12T22:30:44.259Z

As previously announced, the Stanford Research Computing Facility (SRCF), where Sherlock is hosted, will be powered off during the last week of June, in order to safely bring up power to the new SRCF2 datacenter.

Sherlock will not be available for login, to submit jobs or to access files from Saturday June 24th, 2023 at 00:00 PST to Monday July 3rd, 2023 at 18:00 PST.

Jobs will stop running and access to login nodes will be closed at 00:00 PST on Saturday, June 24th, to allow sufficient time for shutdown and pre-downtime maintenance tasks on the cluster, before the power actually goes out. If everything goes according to plan, and barring issues or delays with power availability, access will be restored on Monday, July 3rd at 18:00 PST.

We will use this opportunity to perform necessary maintenance operations on Sherlock that can’t be done while jobs are running, which will avoid having to schedule a whole separate downtime. Sherlock will go offline in advance of the actual electrical shutdown to ensure that all equipment is properly powered off and minimize the risks of disruption and failures when power is restored.

A reservation will be set in the scheduler for the duration of the downtime: if you submit a job on Sherlock and the time you request exceeds the time remaining until the start of the downtime, your job will be queued until the maintenance is over, and the squeue command will report a status of ReqNodeNotAvailable (“Required Node Not Available”).

The hours leading up to a downtime are an excellent time to submit shorter, smaller jobs that can complete before the maintenance begins: as the queues drain there will be many nodes available, and your wait time may be shorter than usual.

As previously mentioned, in anticipation of this week-long downtime, we encourage all users to plan their work accordingly, and ensure that they have contingency plans in place for their computing and data accessibility needs during that time. If you have important data that you need to be able to access while Sherlock is down, we strongly recommend that you start transferring your data to off-site storage systems ahead of time, to avoid last-minute complications. Similarly, if you have deadlines around the time of the shutdown that require computation results, make sure to anticipate those and submit your jobs to the scheduler as early as possible.

We understand that this shutdown will have a significant impact for users who rely on Sherlock for their computing and data processing needs, and we appreciate your cooperation and understanding as we work to improve our Research Computing infrastructure.

For help transferring data, any questions or concerns, please do not hesitate to reach out to [email protected]" rel="noopener nofollow" target="_blank">[email protected].

Instant lightweight GPU instances are now available

2023-04-27T01:05:18.100Z

We know that getting access to GPUs on Sherlock can be difficult and feel a little frustrating at times. Demand has been steadily growing, leading to long pending times, and waiting in line rarely feels great, especially when you have important work to do.

Which is why we are excited to announce the immediate availability of our latest addition to the Sherlock cluster: instant lightweight GPU instances! Every user can now get immediate access to a GPU instance, for a quick debugging session or to explore new ideas in a Notebook.

GPUs are the backbone of high-performance computing. They’ve become an integral component of the toolbox for many users, and are essential for deep learning, scientific simulations, and many other applications. But you don’t always need a full-fledged, top-of-the-line GPU for all your tasks. Sometimes all you want is to run a quick test to prototype an idea, debug a script, or explore new data in an interactive Notebook. For this, the new lightweight GPU instances on Sherlock will give you instant access to a GPU, without having to wait in line and compete with other jobs for resources you don’t need.

Sherlock’s instant lightweight GPU instances leverage NVIDIA’s Multi-Instance GPU (MIG) to provide multiple fully isolated GPU instances on the same physical GPU, each with their own high-bandwidth memory, cache, and compute cores. Those lightweight instances are ideal for small to medium-sized jobs, and lower the barrier to entry for all users

Similar to the interactive sessions available through the dev partition, Sherlock users can now request a GPU via the sh_dev command, and get immediate access with the following command:

$ sh_dev -g 1

For interactive apps in the Sherlock OnDemand interface, requesting a GPU in the dev partition will initiate an interactive session with access to a lightweight GPU instance.

So now, everyone gets a GPU, no questions asked! 😁

We hope those new instances will improve access to GPUs on Sherlock, enable a wider range of use cases, with all the flexibility and performance you need to get your work done, and lead to even more groundbreaking discoveries!

As always, thanks to all of our users for your continuous support and patience as we work to improve Sherlock, and if you have any question or comment, please don’t hesitate to reach out at [email protected]" rel="noopener" target="_blank">[email protected].

A new tool to help optimize job resource requirements

2023-03-25T01:04:21.451Z

It’s not always easy to determine the right amount of resources to request for a computing job. Making sure that the application will have enough resources to run properly, but avoiding over-requests that would make the jobs spend too much time waiting in queue for resources they won’t be using.

To help users inform those choices, we’ve just added a new tool to the module list on Sherlock. ruse is command-line tool developed by Jan Moren which facilitates measuring processes’ resource usage. It periodically measures the resource use of a process and its sub-processes, and can help users find out how much resource to allocate to their jobs. It will determine the actual memory, execution time and cores that individual programs or MPI applications need to request in their job submission options.

You’ll find more information and some examples in the Sherlock documentation at https://www.sherlock.stanford.edu/docs/user-guide/running-jobs/#resource-requests

Hopefully ruse will make it easier to write job resource requests , and allow users to get a better understanding of their applications’ behavior to take better advantage of Sherlock’s capabilities.

As usual, if you have any question or comment, please don’t hesitate to reach out at [email protected]" rel="noopener" target="_blank">[email protected].

SRCF is expanding

2023-02-24T02:00:00Z

The Stanford Research Computing Facility (SRCF), where Sherlock is hosted, has been a highly effective data center since its opening in January of 2014, and demand has grown so much that we’re expanding it! Another identical building (SRCF2) is under construction at SLAC, which will increase our data center capacity when it opens this summer.

In order to bring power to the new building, the entire existing SRCF data center will need to be shut down. The 12kV electrical infrastructure is so pervasive that for the new building to be connected safely, everything needs to be powered off, including the backup generators. It unfortunately means that all servers and equipment will need to be shut down for this event, including Sherlock.

The full building power shutdown is planned for late June 2023, it’s expected to last for about a week, and Sherlock will be unavailable during that time.

During the power outage, Sherlock will be entirely powered down, meaning that it will not allow login or data transfer, the Sherlock OnDemand interface will be down, jobs will not run, and data will not be accessible (including $HOME, $SCRATCH and $OAK). We expect all services to resume normally once power is back up, and jobs that were in queue before the downtime should resume being scheduled normally.

The power outage is currently scheduled for the last week of June 2023. Specific dates and times have not been finalized yet, but we will share more detailed information as the shutdown date gets closer.

In anticipation of this week-long downtime, we encourage all users to plan their work accordingly, and ensure that they have contingency plans in place for their computing and data accessibility needs during that time. If you have important data that you need to be able to access while Sherlock is down, we strongly recommend that you start transferring your data to off-site storage systems ahead of time, to avoid last-minute complications. Similarly, if you have deadlines around the time of the shutdown that require computation results, make sure to anticipate those and submit your jobs to the scheduler as early as possible.

We understand that this shutdown will have a significant impact for users who rely on Sherlock for their computing and data processing needs, and we appreciate your cooperation and understanding as we work to improve our Research Computing infrastructure.

For help in transferring data, any questions or concerns, please do not hesitate to reach out to [email protected]" rel="noopener nofollow" target="_blank">[email protected].

More free compute on Sherlock!

2022-12-14T17:27:18.657Z

We’re thrilled to announce that the free and generally available normal partition on Sherlock is getting an upgrade!

With the addition of 24 brand new SH3_CBASE.1 compute nodes, each featuring one AMD EPYC 7543 Milan 32-core CPU and 256 GB of RAM, Sherlock users now have 768 more CPU cores at there disposal. Those new nodes will complete the existing 154 compute nodes and 4,032 core in that partition, for a new total of 178 nodes and 4,800 CPU cores.

The normal partition is Sherlock’s shared pool of compute nodes, which is available free of charge to all Stanford Faculty members and their research teams, to support their wide range of computing needs.

In addition to this free set of computing resources, Faculty can supplement these shared nodes by purchasing additional compute nodes, and become Sherlock owners. By investing in the cluster, PI groups not only receive exclusive access to the nodes they purchased, but also get access to all of the other owner compute nodes when they're not in use, thus giving them access to the whole breadth of Sherlock resources, currently over over 1,500 compute nodes, 46,000 CPU cores and close to 4 PFLOPS of computing power.

We hope that this new expansion of the normal partition, made possible thanks to additional funding provided by the University Budget Group as part of the FY23 budget cycle, will help support the ever-increasing computing needs of the Stanford research community, and enable even more breakthroughs and discoveries.

As usual, if you have any question or comment, please don’t hesitate to reach out at [email protected]" rel="noopener" target="_blank">[email protected].

ClusterShell on Sherlock

2022-12-03T02:57:22.756Z

Ever wondered how your jobs were doing while they were running? Keeping a eye on a log file is nice, but what if you could quickly gather process lists, usage metrics and other data points from all the nodes your multi-node jobs are running on, all at once?

Enter ClusterShell, the best parallel shell application (and library!) of its kind.

With ClusterShell on Sherlock, you can quickly run a command on all the nodes your job is running on, to gather information about your applications and processes, in real time, and gather live output without having to wait for your job to end to see how it did. And with its tight integration with the job scheduler, no need to fiddle with manual node lists anymore, all it needs is a job id!

You allocated a few nodes in an interactive session and want to distribute some files on each node’s local storage devices? Check: ClusterShell has a copy mode just for this.

Want to double-check that your processes are correctly laid out? Check: you can run a quick command to check the process tree across the nodes allocated to your job with:

$ clush -w @job:$JOBID pstree -au $USER

and verify that all your processes are running correctly.

You’ll find more details and examples in our Sherlock documentation, at https://www.sherlock.stanford.edu/docs/software/using/clustershell

Questions, ideas, or suggestions? Don’t hesitate to reach out to [email protected]" rel="noopener nofollow" target="_blank">[email protected] to let us know!

Job #1, again!

2022-11-06T02:11:25.989Z

This is not the first time, we’ve been through this already (not so long ago, actually) but today, the Slurm job id counter was reset and went from job #67043327 back to job #1.

JobID Partition Start
------------ ---------- -------------------
67043327 normal 2022-11-05T10:18:32
1 normal 2022-11-05T10:18:32

The largest job id that the scheduler can assign on Sherlock is 67,043,327. So when that number is reached, the next submitted job will be assigned job id #1.

This is the second time this job id reset happens in Sherlock’s history, since it debuted in 2014. The first occurrence happened on May 11th, 2020, just a little under 2.5 years ago.

It took about 6 years to submit the first 67 million jobs on Sherlock, but It’s incredible to realize that it took less than half that time to get to that staggering number of submitted jobs once again, and that it all happened since the beginning of the pandemic.

This is an humbling illustration of Sherlock’s central role and its importance to the Stanford research community, especially over the last few months. This give us once again the opportunity to thank each and every one of you, Sherlock users, for your continuous support, your extraordinary motivation and all of your patience and understanding when things break. We’ve never been so proud of supporting your amazing work, especially during those particularly trying times. Stay safe and happy computing!

From Rome to Milan, a Sherlock catalog update

2021-11-30T17:00:00Z

It’s been almost a year and a half since we first introduced Sherlock 3.0 and its major new features: brand new CPU model and manufacturer, 2x faster interconnect, much larger and faster node-local storage, and more! We’ve now reached an inflexion point in Sherlock’s current generation and it’s time to update the hardware configurations available for purchase in the Sherlock catalog.

So today, we’re introducing a new Sherlock catalog refresh, a Sherlock 3.5 of sorts.

The new catalog

So, what changes? What stays the same?
In a nutshell, you’ll continue to be able to purchase the existing node types that you’re already familiar with:

CPU configurations:

CBASE: base configuration ($)
CPERF: high core-count configuration ($$)
CBIGMEM: large-memory configuration ($$$$)

GPU configurations

G4FP32: base GPU configuration ($$)
G4TF64: HPC GPU configuration ($$$)
G8TF64: best-in-class GPU configuration ($$$$)

But they now come with better and faster components!

To avoid confusion, the configuration names in the catalog will be suffixed with a index to indicate the generational refresh, but will keep the same global denomination. For instance, the previous SH3_CBASE configuration is now replaced with a SH3_CBASE.1 configuration that still offers 32 CPU cores and 256 GB of RAM.

A new CPU generation

The main change in the existing configuration is the introduction of the new AMD 3rd Gen EPYC Milan CPUs. In addition to the advantages of the previous Rome CPUs, this new generation brings:

a new micro-architecture (Zen3)
a ~20% performance increase in instructions completed per clock cycle (IPC)
enhanced memory performance, with a unified 32 MB L3 cache
improved CPU clock speeds

More specifically, for Sherlock, the following CPU models are now used:

Model	Sherlock 3.0 (Rome)	Sherlock 3.5 (Milan)
`CBASE`	1× 7502 (32-core, 2.50GHz)	1× 7543 (32-core, 2.75GHz)
`CPERF`	2× 7742 (64-core, 2.25GHz)	2× 7763 (64-core, 2.45GHz)
`CBIGMEM`	2× 7502 (32-core, 2.50GHz)	2× 7543 (32-core, 2.75GHz)
`G4FP32`	1× 7502 (32-core, 2.50GHz)	1× 7543 (32-core, 2.75GHz)
`G4TF64`	2× 7502 (32-core, 2.50GHz)	2× 7543 (32-core, 2.75GHz)
`G8TF64`	2× 7742 (64-core, 2.25GHz)	2× 7763 (64-core, 2.45GHz)

In addition to IPC and L3 cache improvements, the new CPUs also bring a frequency boost that will provide a substantial performance improvement.

New GPU options

On the GPU front, the two main changes are the re-introduction of the G4FP32 model, and the doubling of GPU memory all across the board.

GPU memory is quickly becoming the constraining factor for training deep-learning models that keep increasing in size. Having large amounts of GPU memory is now key for running medical imaging workflows, computer vision models, or anything that requires processing large images.

The entry-level G4FP32 model is back in the catalog, with a new NVIDIA A40 GPU in an updated SH3_G4FP32.2 configuration. The A40 GPU not only provides higher performance than the previous model it replaces, but it also comes with twice as much GPU memory, with a whopping 48GB of GDDR6.

The higher-end G4TF64 and G8TF64 models have also been updated with newer AMD CPUs, as well as updated versions of the NVIDIA A100 GPU, now each featuring a massive 80GB of HBM2e memory.

Get yours today!

For more details and pricing, please check out the Sherlock catalog (SUNet ID required).

If you’re interested in getting your own compute nodes on Sherlock, all the new configurations are available for purchase today, and can be ordered online though the Sherlock order form (SUNet ID required).

As usual, please don’t hesitate to [email protected]" rel="noopener" target="_blank">reach out if you have any questions!