Sherlock changelog

A brand new Sherlock OnDemand experience

2023-11-16T02:00:00Z

Following a long tradition of announces and releases during the SuperComputing conference, and while SC23 is underway in Denver CO, Stanford Research Computing is proud to unveil Sherlock OnDemand 3.0, a cutting-edge enhancement to its computing and data storage resources, revolutionizing user interaction and efficiency.

The upgraded Sherlock OnDemand is available immediately at https://ondemand.sherlock.stanford.edu

This new release brings a host of transformative changes. A lot happened under the hood, but the visible changes are significant as well.

Infrastructure upgrades:

A new URL: Sherlock OnDemand is now accessible at https://ondemand.sherlock.stanford.edu, in line with our other instances, for a more homogeneous user experience across Research Computing systems. The previous URL will still work for a time, and redirections will be progressively deployed to ease the transition.
New engine, same feel: a lot of internal components have undergone substantial updates, but the familiar interface remains intact, ensuring a seamless transition for existing users.
Streamlined authentication: Sherlock OnDemand now uses OIDC via the Stanford central Identity Provider instead of SAML, resulting in a lighter, more robust configuration for enhanced security.
Enhanced Performance: expect a more responsive interface and improved reliability with the eradication of 422 HTTP errors.

User-centric features:

Expanded file access: all your Oak groups, are now conveniently listed in the embedded file browser for easier and more comprehensive access to your data. And if you have rclone remotes already configured on Sherlock, you’'ll find them there as well!
Effortless support tickets: you can now send support tickets directly from the OnDemand interface, which will automatically include contextual information about your interactive sessions, to simply issue resolution.
New interactive apps: In addition to the existing apps, VS Code server, MATLAB, and JupyterLab join the platform, offering expanded functionalities, like the ability of loading and unloading of modules within JupyterLab directly.
Yes, you read that right: we now have VS Code and MATLAB in Sherlock OnDemand!
The RStudio app has also been rebuilt from the ground up, providing a much better and reliable experience.
Customizable working directories: users can now select a working directory across all interactive apps, for easier customization of their work environment.

For more details and guidance on using the new features, check out the updated documentation at https://www.sherlock.stanford.edu/docs/user-guide/ondemand/.

This update delivers a brand new computing experience, designed to empower you in your work. Sherlock OnDemand 3.0 marks a significant milestone in optimizing user access to computing resources, lowering the barrier to entry for new users, and empowering researchers with an unparalleled computing environment. We're excited to see how it will enhance your productivity and efficiency, so dive into this transformative experience today and elevate your computing endeavors to new heights with Sherlock OnDemand 3.0!

And as usual, if you have any question, comment or suggestion, don’t hesitate to reach out at [email protected]" rel="noopener nofollow" target="_blank" title="support">[email protected].

Final hours announced for the June 2023 SRCF downtime

2023-05-12T22:30:44.259Z

As previously announced, the Stanford Research Computing Facility (SRCF), where Sherlock is hosted, will be powered off during the last week of June, in order to safely bring up power to the new SRCF2 datacenter.

Sherlock will not be available for login, to submit jobs or to access files from Saturday June 24th, 2023 at 00:00 PST to Monday July 3rd, 2023 at 18:00 PST.

Jobs will stop running and access to login nodes will be closed at 00:00 PST on Saturday, June 24th, to allow sufficient time for shutdown and pre-downtime maintenance tasks on the cluster, before the power actually goes out. If everything goes according to plan, and barring issues or delays with power availability, access will be restored on Monday, July 3rd at 18:00 PST.

We will use this opportunity to perform necessary maintenance operations on Sherlock that can’t be done while jobs are running, which will avoid having to schedule a whole separate downtime. Sherlock will go offline in advance of the actual electrical shutdown to ensure that all equipment is properly powered off and minimize the risks of disruption and failures when power is restored.

A reservation will be set in the scheduler for the duration of the downtime: if you submit a job on Sherlock and the time you request exceeds the time remaining until the start of the downtime, your job will be queued until the maintenance is over, and the squeue command will report a status of ReqNodeNotAvailable (“Required Node Not Available”).

The hours leading up to a downtime are an excellent time to submit shorter, smaller jobs that can complete before the maintenance begins: as the queues drain there will be many nodes available, and your wait time may be shorter than usual.

As previously mentioned, in anticipation of this week-long downtime, we encourage all users to plan their work accordingly, and ensure that they have contingency plans in place for their computing and data accessibility needs during that time. If you have important data that you need to be able to access while Sherlock is down, we strongly recommend that you start transferring your data to off-site storage systems ahead of time, to avoid last-minute complications. Similarly, if you have deadlines around the time of the shutdown that require computation results, make sure to anticipate those and submit your jobs to the scheduler as early as possible.

We understand that this shutdown will have a significant impact for users who rely on Sherlock for their computing and data processing needs, and we appreciate your cooperation and understanding as we work to improve our Research Computing infrastructure.

For help transferring data, any questions or concerns, please do not hesitate to reach out to [email protected]" rel="noopener nofollow" target="_blank">[email protected].

More free compute on Sherlock!

2022-12-14T17:27:18.657Z

We’re thrilled to announce that the free and generally available normal partition on Sherlock is getting an upgrade!

With the addition of 24 brand new SH3_CBASE.1 compute nodes, each featuring one AMD EPYC 7543 Milan 32-core CPU and 256 GB of RAM, Sherlock users now have 768 more CPU cores at there disposal. Those new nodes will complete the existing 154 compute nodes and 4,032 core in that partition, for a new total of 178 nodes and 4,800 CPU cores.

The normal partition is Sherlock’s shared pool of compute nodes, which is available free of charge to all Stanford Faculty members and their research teams, to support their wide range of computing needs.

In addition to this free set of computing resources, Faculty can supplement these shared nodes by purchasing additional compute nodes, and become Sherlock owners. By investing in the cluster, PI groups not only receive exclusive access to the nodes they purchased, but also get access to all of the other owner compute nodes when they're not in use, thus giving them access to the whole breadth of Sherlock resources, currently over over 1,500 compute nodes, 46,000 CPU cores and close to 4 PFLOPS of computing power.

We hope that this new expansion of the normal partition, made possible thanks to additional funding provided by the University Budget Group as part of the FY23 budget cycle, will help support the ever-increasing computing needs of the Stanford research community, and enable even more breakthroughs and discoveries.

As usual, if you have any question or comment, please don’t hesitate to reach out at [email protected]" rel="noopener" target="_blank">[email protected].

From Rome to Milan, a Sherlock catalog update

2021-11-30T17:00:00Z

It’s been almost a year and a half since we first introduced Sherlock 3.0 and its major new features: brand new CPU model and manufacturer, 2x faster interconnect, much larger and faster node-local storage, and more! We’ve now reached an inflexion point in Sherlock’s current generation and it’s time to update the hardware configurations available for purchase in the Sherlock catalog.

So today, we’re introducing a new Sherlock catalog refresh, a Sherlock 3.5 of sorts.

The new catalog

So, what changes? What stays the same?
In a nutshell, you’ll continue to be able to purchase the existing node types that you’re already familiar with:

CPU configurations:

CBASE: base configuration ($)
CPERF: high core-count configuration ($$)
CBIGMEM: large-memory configuration ($$$$)

GPU configurations

G4FP32: base GPU configuration ($$)
G4TF64: HPC GPU configuration ($$$)
G8TF64: best-in-class GPU configuration ($$$$)

But they now come with better and faster components!

To avoid confusion, the configuration names in the catalog will be suffixed with a index to indicate the generational refresh, but will keep the same global denomination. For instance, the previous SH3_CBASE configuration is now replaced with a SH3_CBASE.1 configuration that still offers 32 CPU cores and 256 GB of RAM.

A new CPU generation

The main change in the existing configuration is the introduction of the new AMD 3rd Gen EPYC Milan CPUs. In addition to the advantages of the previous Rome CPUs, this new generation brings:

a new micro-architecture (Zen3)
a ~20% performance increase in instructions completed per clock cycle (IPC)
enhanced memory performance, with a unified 32 MB L3 cache
improved CPU clock speeds

More specifically, for Sherlock, the following CPU models are now used:

Model	Sherlock 3.0 (Rome)	Sherlock 3.5 (Milan)
`CBASE`	1× 7502 (32-core, 2.50GHz)	1× 7543 (32-core, 2.75GHz)
`CPERF`	2× 7742 (64-core, 2.25GHz)	2× 7763 (64-core, 2.45GHz)
`CBIGMEM`	2× 7502 (32-core, 2.50GHz)	2× 7543 (32-core, 2.75GHz)
`G4FP32`	1× 7502 (32-core, 2.50GHz)	1× 7543 (32-core, 2.75GHz)
`G4TF64`	2× 7502 (32-core, 2.50GHz)	2× 7543 (32-core, 2.75GHz)
`G8TF64`	2× 7742 (64-core, 2.25GHz)	2× 7763 (64-core, 2.45GHz)

In addition to IPC and L3 cache improvements, the new CPUs also bring a frequency boost that will provide a substantial performance improvement.

New GPU options

On the GPU front, the two main changes are the re-introduction of the G4FP32 model, and the doubling of GPU memory all across the board.

GPU memory is quickly becoming the constraining factor for training deep-learning models that keep increasing in size. Having large amounts of GPU memory is now key for running medical imaging workflows, computer vision models, or anything that requires processing large images.

The entry-level G4FP32 model is back in the catalog, with a new NVIDIA A40 GPU in an updated SH3_G4FP32.2 configuration. The A40 GPU not only provides higher performance than the previous model it replaces, but it also comes with twice as much GPU memory, with a whopping 48GB of GDDR6.

The higher-end G4TF64 and G8TF64 models have also been updated with newer AMD CPUs, as well as updated versions of the NVIDIA A100 GPU, now each featuring a massive 80GB of HBM2e memory.

Get yours today!

For more details and pricing, please check out the Sherlock catalog (SUNet ID required).

If you’re interested in getting your own compute nodes on Sherlock, all the new configurations are available for purchase today, and can be ordered online though the Sherlock order form (SUNet ID required).

As usual, please don’t hesitate to [email protected]" rel="noopener" target="_blank">reach out if you have any questions!

SH3_G4FP32 nodes are back in the catalog!

2020-11-05T01:49:00.001Z

A new GPU option is available in the Sherlock catalog… again!

After a period of unavailability and a transition between GPU generations, where previous models were retired while new ones were not available yet, we’re pleased to announce that the entry-level GPU node configuration is now back in the catalog. With a vengeance!

Built around the same platform as the previous SH3_G4FP32 generation, the new SH3_G4FP32.1 model features:

32 CPU cores
256 GB of memory
2TB of local NVMe scratch space
4x GeForce RTX 3090 GPUs, each featuring 24GB of GPU memory
a 200GB/s Infiniband HDR interface

Particularly well-suited for applications that don’t require full double-precision computations (FP64), the top-of-the-line RTX 3090 GPU is based on the latest NVIDIA Ampere architecture and provides what’s probably the best performance/cost ratio on the market today for those use cases, and delivers almost twice the performance of the previous generations on many ML/AI workloads, as well as a significant boost for Molecular Dynamics and CryoEM applications.

For more details and pricing, please check out the Sherlock catalog (SUNet ID required), and if you’re interested in purchasing your own compute nodes for Sherlock, the new SH3_G4FP32.1 configuration is available for purchase today, and can be ordered online though the Sherlock order form (SUNet ID required).

Sherlock 3.0 is here!

2020-08-20T00:42:00.001Z

It’s been a long, long, way too long of a wait, but despite a global pandemic, heatwaves, thunderstorms, power shutoffs, fires and smoke, it’s finally here!

Today, we’re very excited to announce the immediate availability of Sherlock 3.0, the third generation of the Sherlock cluster.

What is Sherlock 3.0?

First, let’s take a quick step back for context.

The Sherlock cluster is built around core Infiniband fabrics, which connect compute nodes together and allow them to work as a single entity. As we expand Sherlock over time, more compute nodes are added to the cluster, and when a core fabric reaches capacity, a new one needs to be spun up. This is usually a good opportunity to refresh the compute node hardware characteristics, as well as continue expanding and renewing ancillary equipment and services, such as login nodes, DTNs, storage systems, etc. The collection of compute and service nodes connected to the same Infiniband fabric constitutes a sort of island, or generation, that could live on its own, but is actually an integral part of the greater, unified Sherlock cluster.

So far, since its inception in 2014, Sherlock has grown over two generations of nodes: the first one built around an FDR (56Gb/s) Infiniband fabric, and the second one, started in 2017, around an EDR (100Gb/s) fabric.

Late last year, that last EDR fabric reached capacity, and after a long and multifactorial hiatus, today, we’re introducing the third generation of Sherlock, architectured around a new Infiniband fabric, and a completely refreshed compute node offering.

What does it look like?

Sherlock still looks like a bunch of black boxes with tiny lights, stuffed in racks 6ft high, and with an insane number of cables going everywhere.

But in more technical details, Sherlock 3.0 features:

a new, faster interconnect | Infiniband HDR, 200Gb/s
The new interconnect provides more bandwidth and lower latency to all the new nodes on Sherlock, for either inter-node communication in large parallel MPI applications, or for accessing the $SCRATCH and $OAK parallel file systems.
Sherlock is one of the first HPC clusters in the world to provide 200Gb/s to the nodes.
new and faster processors | AMD 2nd generation EPYC (Rome) CPUs
To take advantage of the doubled inter-node bandwidth, a brand new generation of CPUs was required, to provide enough internal bandwidth between the CPUs and the network interfaces. The AMD Rome CPUs are actually the first (and currently still the only) x86 CPU model to provide PCIe Gen4 connectivity, that enables faster local and remote I/O, and that can unlock 200Gb/s network speeds.
Those CPUs are also faster, draw less power, and provide more cores per socket than the ones found in the previous generations of Sherlock nodes, with a minimum of 32 CPU cores per node.
more (and faster) internal storage | 2TB NVMe per node
Sherlock 3.0 nodes now each feature a minimum of 2TB of local NVMe storage (over 10x the previous amount), for applications that are particularly sensitive to IOPS rates.
refreshed $HOME storage
More nodes means more computing power, but it also means more strain on the shared infrastructure. To absorb it, we’ve also refreshed and expanded the storage cluster that supports the $HOME and $GROUP_HOME storage spaces, to provide higher bandwidth, more IOPS, and better availability.
more (and faster) login and DTN nodes
Sherlock 3.0 also feature 8 brand new login nodes, that are part of the login.sherlock.stanford.edu login pool, and each feature a pair of AMD 7502 CPUs (for a total of 64 cores) and 512 GB of RAM. As well as a new pair of dedicated Data Transfer Nodes (DTNs)
refreshed and improved infrastructure
The list would be too long to go through exhaustively, but between additional service nodes to better scale the distributed cluster management infrastructure, improved Ethernet topology between the racks, and a refreshed hardware framework for the job scheduler, all the aspects of Sherlock have been rethought and improved.

What does it change for me?

In terms of habits and workflows: nothing. You don’t have to change anything and can continue to use Sherlock exactly the way you’ve been using it so far.

Sherlock is still a single cluster, with the same:

single point of entry at login.sherlock.stanford.edu,
single and ubiquitous data storage space (you can still access all of your data on all the file systems, from all the nodes in the cluster),
single application stack (you can load the same module and run the same software on all Sherlock nodes).

But it now features a third island, with a new family of compute nodes.

One thing you’ll probably notice pretty quickly is that your pending times in queue for the normal, bigmem and gpu partitions have been dropping. Considerably.

This is because, thanks to the generous sponsorship of the Stanford Provost, we’ve been able to add the following resources to Sherlock’s public partitions:

partition	#nodes	node specs
`normal`	72	32-core (1x 7502) w/ 256GB RAM
`normal`	2	128-core (2x 7742) w/ 1TB RAM
`bigmem`	1	64-core (2x 7502) w/ 4TB RAM
`gpu`	16	32-core (1x 7502P) w/ 256GB RAM and 4x RTX 2080 Ti GPUs
`gpu`	2	32-core (1x 7502P) w/ 256GB RAM and 4x V100S GPUs
Total	93	3,200 cores, 30TB RAM, 72 GPUs

Those new Sherlock 3.0 nodes are adding over twice the existing computing power available for free to every Sherlock user in the normal, bigmem and gpu partitions.

How can I use the new nodes?

It’s easy! You can keep submitting your jobs as usual, and the scheduler will automatically try to pick the new nodes that satisfy your request requirements if they’re available.

If you want to target the new nodes specifically, take a look at the output of sh_node_feat: all the new nodes have features defined that allow the scheduler to specifically select them when your job requests particular constraints.

For instance, if you want to select nodes:

with HDR IB connectivity, you can use -C IB:HDR
with AMD Rome CPUs, you can use -C CPU_GEN:RME
with 7742 CPUs, you can use -C CPU_SKU:7742
with Turing GPUs, you can use -C GPU_GEN:TUR

Can I get more of it?

Absolutely! And we’re ready to take orders today.

If you’re interested in getting your own compute nodes on Sherlock, we’ve assembled a catalog of select configurations that you can choose from, and worked very hard with our vendors to maintain comparable price ranges with our previous generation offerings.

You’ll find the detailed configuration and pricing in the Sherlock Compute Nodes Catalog, and we’ve also prepared an Order Form that you can use to provide the required information to purchase those nodes

Sherlock catalog
http://www.sherlock.stanford.edu/docs/overview/orders/catalog
Order form
http://www.sherlock.stanford.edu/docs/overview/orders/form

For complete details about the purchasing process, please take a look at
https://www.sherlock.stanford.edu/docs/overview/orders/ and as usual,
please let us know if you have any questions.

Finally, we wanted to sincerely thank every one of you for your patience while we were working on bringing up this new cluster generation, in an unexpectedly complicated global context. We know it’s been a very long wait, but hopefully it will have been worthwhile.

Happy computing and don’t hesitate to [email protected]" target="_blank" rel="noopener">reach out!

Oh, and Sherlock is on Slack now, so feel free to come join us there too!

Sherlock joins the fight against COVID-19

2020-03-16T17:06:00.001Z

At SRCC, we’ve been monitoring the COVID-19 situation very closely, and we are following all of the University guidance and recommendations^[1] regarding social distancing and workplace adaptations. While we’re continuing to operate our services normally^[2], we’ve implemented changes to minimize health risks: we’ve moved our office hours sessions online, and started online consultation appointments.

But we wanted to do more, mobilize our forces and try to contribute with the resources we have, at our own level.

This is why, starting immediately, we’ll be dedicating a portion of Sherlock’s resources to research projects related to COVID-19. If you’re working on such a project and need additional resources on Sherlock, please contact us at [email protected]" target="_blank" rel="noopener">[email protected], with a quick description of your work, and we’ll coordinate access to those resources for you and your research team.

We would also like to call on Sherlock owners: if you have dedicated compute nodes on Sherlock, and would like to contribute some of those resources to this essential computing effort, please reach out!

Our mission is to support research, and with this initiative, we hope Sherlock will help time-critical projects to get to results faster, and contribute to the fight against this coronavirus.

We strongly encourage everyone to regularly consult and follow recommendations provided on https://healthalerts.stanford.edu, especially the latest Guidance for the research environment. ↩
for more information about SRCC’s continuity of support, please see https://srcc.stanford.edu/news/srcc-continuity-support-during-covid-19. ↩

More scratch space for everyone!

2019-12-03T23:30:00.001Z

Today, we’re super excited to announce several major changes to the /scratch filesystem on Sherlock.

What’s `/scratch` already?

/scratch is Sherlock’s temporary, parallel and high-performance filesystem. It’s available from all the compute nodes in the cluster, and is aimed at storing temporary data, like raw job output, intermediate files, or unprocessed results.

All the details about /scratch can be found in the Sherlock documentation, at https://www.sherlock.stanford.edu/docs/storage/filesystems/#scratch

A brand new storage system

So first of all, Sherlock’s /scratch now uses a brand new underlying storage system: it’s newer, faster and better that the old system in many ways that are described in much more details in this other post.

But to sum it up, using newer and faster hardware, the new /scratch storage system is twice as large, dramatically accelerate small files access and metadata operations, and enables new filesystem features for better overall performance.

If you’d like to take advantage of that new system and are wondering what you need to benefit from its improved performance, the answer is pretty simple: nothing! Your data is already there: if you’re using $SCRATCH or $GROUP_SCRATCH today, you don’t have to do anything, you’re already using the new storage system.

How did that happen? You can read all about it in that post I mentioned above.

More space for everyone!

Now, some things don’t change, but others do. We’re really excited to announce that starting today, every user on Sherlock gets ~~twice~~ ~~thrice~~ 🎉✨ five times✨🎈 the amount of storage that was previously offered.

Yep, that’s right, starting today, every user on Sherlock gets 100TB in $SCRATCH. And because sharing is caring, each group gets an additional 100TB to share data in $GROUP_SCRATCH.

But wait, there’s more.

Because we know ownership-based user and group quotas were confusing at times, we’re moving away from them and are adopting a new, directory-based quota system. That means that all the files that are under a given $SCRATCH directory, and only them, will be accounted for in the quota usage, no matter which user and group owns them. It will makes finding files that count towards a given quota much easier.

Previously, with ownership-based accounting, a user with data in both her own $SCRATCH folder and in $GROUP_SCRATCH would see the sum of all those files’ size counted against both her user quota and the group quota. Plus, the group quota was de facto acting as a cap for all the users in the same group, which was penalizing for groups with more members.

Now, data in a user’s $SCRATCH and $GROUP_SCRATCH are accounted for independently, and they’re cumulative. Meaning that no matter how many members a group counts, each user will be able to use the same amount of storage, and won’t be impacted by what others in the group use.

Here what things looks like, more visually (and to scale!):

before, individual ownership-based user quota (in blue) were limited by the overarching group quota (in purple).
now, each user can use up to their quota limit, without being impacted by others, and an additional 100TB is available for the group to share data among group members.

So not only individual quota values have been increased, but the change in quota type also means that the cumulative usable space in /scratch by each group will be much larger than before.

A new retention period

With that increase in space, we’re also updating the retention period on /scratch to 90 days. And because we don’t want to affect files that have been created less than 3 months ago, this change will not take effect immediately.

Starting Feb.3, 2020, all files stored in /scratch that have not been modified in the last 90 days will automatically be deleted from the filesystem.

This is in alignment with the vast majority of other computing centers, and a way to emphasize the temporary nature of the filesystem: /scratch is really designed to store temporary data, and provide high-performance throughput for parallel I/O.

For long-term storage of research data, we always recommend using Oak, which is also directly available from all compute nodes on Sherlock (you’ll find all the details about Oak at https://oak-storage.stanford.edu). Data can freely be moved between /scratch and Oak at very high throughput rates. We can suggest optimized solutions for this, so please don’t hesitate to reach out if you have any question.

TL;DR

Today, we’re announcing:

a brand new storage system for /scratch on Sherlock
a quota increase to 100TB for each user in $SCRATCH and each group in $GROUP_SCRATCH
the move to directory-based quotas for easier accounting of space utilization, and for allowing each user to reach their $SCRATCH quota
a new 90-day retention period for all files in /scratch, starting Feb. 3, 2020

All those changes have been reflected in the documentation at https://www.sherlock.stanford.edu/docs/storage/filesystems/

We hope those changes will enable more possibilities for computing on Sherlock, by allowing storage of larger datasets and running larger simulations.

As usual, if you have any question or comment, please don’t hesitate to let us know at [email protected]" target="_blank" rel="noopener">[email protected].

Call for scientific awesomeness!

2019-10-21T22:24:00.001Z

Every fall for the last 30 years, the HPC community has gathered at SC, the International Conference for High Performance Computing Networking, Storage, and Analysis, and will meet again this year, for the SC19 conference, in Denver, CO (Nov. 17-22).

Stanford Research Computing and SLAC will be at the conference, to exchange with our peers at other institutions, meet vendors and present our activities in support of the Stanford/SLAC research community.

This is always an excellent opportunity to present real-world utilization of our computing resources and showcase amazing research projects that those resources enable.

If you’d like to have your work featured in the Stanford/SLAC booth, and would be willing to provide visual content (pictures or videos), please feel free contact us at [email protected]" target="_blank" rel="noopener">[email protected] with a few words about your research projects that were enabled by having computing and/or storage platforms at Stanford available to you. We’ll be very happy to showcase your work during this international conference and show the world what you all do.

Out with the old, in with the new!

2019-03-20T00:21:00.001Z

It’s been a little over five years since Sherlock first entered production and ran its first jobs. A lot of time has passed, and with it, Sherlock grew ten-fold, its user base exploded and technology trends evolved significantly.

The old

The original Sherlock nodes are still running jobs, but they’re incrementally moving towards irrelevance as they lack the features, the performance and the power efficiency of the most recently added compute nodes.

This is why we’re starting the process of gradually retiring the original Sherlock nodes. This will mainly affect the Sherlock public tiers in the normal, gpu and bigmem partition, but also some of the earliest owner partitions (we’ve started contacting the owners of those nodes).

The most noticeable effect will be that the size of those partitions will start shrinking, and over time, those nodes and their 5-year old Ivy Bridge generation CPUs will progressively disappear.

To compensate for these retirements, we’re also introducing new nodes to the normal partition.

The new

The new nodes are based on the latest Intel Skylake microarchitecture and have the following specifications:

dual-socket Intel Xeon Gold 5118 (2x 12-core, 24 core per node)
192GB of memory (RAM)
200GB local SSD (for $L_SCRATCH)
EDR Infiniband (100Gbps) connectivity to the Sherlock fabric.

You’ll be able to request those nodes specifically by using job submission constraints and adding the following flag to your job submission options: -C "CPU_GEN:SKX"

To see the list of all the available node features and characteristics that can be requested in the normal partition, you can run:

$ sh_node_feat -p normal
CPU_FRQ:2.30GHz
CPU_FRQ:2.40GHz
CPU_FRQ:2.60GHz
CPU_GEN:BDW
CPU_GEN:HSW
CPU_GEN:IVB
CPU_GEN:SKX
CPU_SKU:5118
CPU_SKU:E5-2640v3
CPU_SKU:E5-2640v4
CPU_SKU:E5-2650v2

And as always, please don’t hesitate to let us know if you have any question, by reaching out to [email protected]" target="_blank" rel="noopener">[email protected].