Sherlock changelog

Job #1, again!

2022-11-06T02:11:25.989Z

This is not the first time, we’ve been through this already (not so long ago, actually) but today, the Slurm job id counter was reset and went from job #67043327 back to job #1.

JobID Partition Start
------------ ---------- -------------------
67043327 normal 2022-11-05T10:18:32
1 normal 2022-11-05T10:18:32

The largest job id that the scheduler can assign on Sherlock is 67,043,327. So when that number is reached, the next submitted job will be assigned job id #1.

This is the second time this job id reset happens in Sherlock’s history, since it debuted in 2014. The first occurrence happened on May 11th, 2020, just a little under 2.5 years ago.

It took about 6 years to submit the first 67 million jobs on Sherlock, but It’s incredible to realize that it took less than half that time to get to that staggering number of submitted jobs once again, and that it all happened since the beginning of the pandemic.

This is an humbling illustration of Sherlock’s central role and its importance to the Stanford research community, especially over the last few months. This give us once again the opportunity to thank each and every one of you, Sherlock users, for your continuous support, your extraordinary motivation and all of your patience and understanding when things break. We’ve never been so proud of supporting your amazing work, especially during those particularly trying times. Stay safe and happy computing!

3.3 PFlops: Sherlock hits expansion milestone

2021-04-03T00:00:00Z

Sherlock is a traditional High-Performance Computing cluster in many aspects. But unlike most of similarly-sized clusters where hardware is purchased all at once, and refreshed every few years, it is in constant evolution. Almost like a living organism, it changes all the time: mostly expanding as individual PIs, research groups, labs and even whole Schools contribute computing resources to the system ; but also sometimes contracting, when older equipment is retired.

A significant expansion milestone

A few days ago, Sherlock has reached a major expansion milestone, largely owing to significant purchases from the School of Earth, Energy & Environmental Sciences, but also thanks to multiple existing owner groups who decided to renew their investment in Sherlock by purchasing additional hardware.

With these recent additions, Sherlock reached a theoretical power of over 3 Petaflops, 3 thousand million million (10¹⁵) floating-point operations per second. That would place it around the 150th position in the most recent TOP500 list of the most powerful computer systems in the world.

Among the newly added nodes, a number of SH3_G8TF64 nodes, each featuring 128 CPU cores, 1TB of RAM, 8x A100 SXM4 GPUs (NVLink) and two Infiniband HDR interfaces providing 400Gb/s of interconnect bandwidth, both for storage and inter-node communication. Those nodes alone provide over half a Petaflop of computing power.

Sherlock now features over 1,700 compute nodes, occupying 45 data-center racks, and consuming close to half a megawatt of power. Over 44,000 CPU cores, more than 120 Infiniband switches and close to 20 miles of cables help support the daily computing activities of over 5,000 users. For even more facts and numbers, checkout the Sherlock Facts page!

A steady growth

Since in first days in 2014, and its initial 120 nodes, Sherlock has been growing at a steady pace. Three generations and as many Infiniband fabrics later, and after a few months of slowdown at the beginning of 2020, expansion has resumed and is going stronger than ever:

The road ahead

To keep expanding Sherlock and continue to serve the computing needs of the Stanford research community, rack space used by first generation Sherlock nodes needs to be reclaimed to make room for the next generation. Those 1st-gen nodes have been running well over their initial service life of 4 years, and in most cases, we’ve even been able to keep them running for an extra year. But data-center space being the hot property it has now become, and since demand for new nodes is not exactly dwindling down, we’ll be starting to retire the older Sherlock nodes to accommodate the ever-increasing requests for more computing power. We’ve started working on renewal plans with those node owners, and the process is already underway.

So for a while, Sherlock will shrink in size, as old nodes are retired. Before it can start growing again!

Catalog changes

As we move forward, the Sherlock Compute Nodes Catalog is also evolving, to follow the latest technological trends, and to adapt to the computing needs of our research community.

As part of this evolution, the recently announced SH3_G4FP32 configuration is sadly not available anymore, as vendors suddenly and globally discontinued the consumer-grade GPU model that was powering this configuration. They don’t have plans to bring back anything comparable, so that configuration had to be pulled from the catalog, unfortunately.

On a more positive note, a significant and exciting catalog refresh is coming up, and will be announced soon. Stay tuned! 🤫

As usual, we want to sincerely thank every one of you, Sherlock users, for your patience when things break, your extraordinary motivation and your continuous support. We’re proud of supporting your amazing work, and Sherlock simply wouldn’t exist without you.

Happy computing and don’t hesitate to [email protected]" rel="noopener" target="_blank">reach out if you have any questions!

Job #1

2020-05-11T20:09:00.001Z

If you’ve been submitting jobs on Sherlock over the last couple days, you probably noticed something different about your your job ids… They lost a couple digits!

If you submitted a job last week, its job id was likely in the 67,000,000s. Today, it’s back in the 100,000s. What happened? Did we reset anything? Did we start simplifying job ids because there were too many numbers to keep track of?

Not really.

It’s just that so many jobs are submitted to Sherlock these days (and even more so since the beginning of the stay-at-home directives), that we reached the maximum job id that the scheduler can use.

Those job ids are roughly 26 bits in length, with a little headroom for special cases, and the largest job id that the scheduler can assign on Sherlock is 67,043,327. It means that when that number is reached, the next submitted job will be assigned jobid #1.

Both were submitted on Friday night, and started running Saturday morning:

 JobID Partition Submit Start
------------ ---------- ------------------- -------------------
1 normal 2020-05-08T22:21:28 2020-05-09T06:04:50
67043327 normal 2020-05-08T22:21:28 2020-05-09T06:05:16

A few months ago, we celebrated job #50,000,000. Today, we’re celebrating job #1, the beginning of a new cycle. :)

Thanks to each and every one of you, Sherlock users, for your continuous support, your extraordinary motivation and all of your patience and understanding when things break. We’ve never been so proud of supporting your amazing work, especially during those particularly trying times. Stay safe and happy computing!

🎉 Job #50,000,000!

2019-09-12T17:44:00.001Z

We just wanted to share that Sherlock recently ran job #50,000,000! 🎈🎉

This is a significant milestone since Sherlock, in its current form^[1], started running its first job in January 2017. Fifty million jobs in less than 3 years is no small feat, and it wouldn’t have been achieved without the trust and confidence of all of our users.

Thanks to each and every one of you, Sherlock users, for your continuous support, your extraordinary motivation and all of your patience during times when things break. We’re proud of supporting your amazing work, and Sherlock simply wouldn’t exist without you.

And also, kudos to Xiang Zhu, from Prof. Wing Wong's group in the Department of Statistics for submitting job #50,000,000, you won a $50 Coupa card! ☕

Sherlock was born in 2014 and its first instance ran close to 30 million jobs between 2014 and 2017. It was reborn under its current Sherlock 2.0 form in 2017. ↩