Sherlock changelog

ClusterShell on Sherlock

2022-12-03T02:57:22.756Z

Ever wondered how your jobs were doing while they were running? Keeping a eye on a log file is nice, but what if you could quickly gather process lists, usage metrics and other data points from all the nodes your multi-node jobs are running on, all at once?

Enter ClusterShell, the best parallel shell application (and library!) of its kind.

With ClusterShell on Sherlock, you can quickly run a command on all the nodes your job is running on, to gather information about your applications and processes, in real time, and gather live output without having to wait for your job to end to see how it did. And with its tight integration with the job scheduler, no need to fiddle with manual node lists anymore, all it needs is a job id!

You allocated a few nodes in an interactive session and want to distribute some files on each node’s local storage devices? Check: ClusterShell has a copy mode just for this.

Want to double-check that your processes are correctly laid out? Check: you can run a quick command to check the process tree across the nodes allocated to your job with:

$ clush -w @job:$JOBID pstree -au $USER

and verify that all your processes are running correctly.

You’ll find more details and examples in our Sherlock documentation, at https://www.sherlock.stanford.edu/docs/software/using/clustershell

Questions, ideas, or suggestions? Don’t hesitate to reach out to [email protected]" rel="noopener nofollow" target="_blank">[email protected] to let us know!

Keep up to date with software updates

2021-10-29T17:29:35.955Z

To help users stay on top of software changes on Sherlock, we’ve recently introduced a new software updates RSS feed.

It’s available from the Sherlock software list page, and you can directly add it to your RSS reader of choice.

And if you’re already following our Slack channels, you can subscribe to the new #sherlock-software-updates channel to get automatically notified of new software updates on Sherlock, directly from Slack.

Happy computing! And as usual, please feel free to [email protected]" rel="noopener" target="_blank">reach out if you have comments or questions.

Secure TensorBoard sessions with Sherlock OnDemand

2019-12-13T18:29:00.001Z

If you’re into machine learning (and who isn’t these days?), you probably know all about TensorBoard already.

If you don’t, TensorBoard is TensorFlow's visualization toolkit. It provides the visualization and tooling needed for machine learning workflows: it enables tracking experiment metrics like loss and accuracy, visualizing model graphs, viewing histograms of weights, biases, or other tensors as they change over time, or profiling TensorFlow programs. All kinds of cool and useful stuff.

TensorBoard (lack of) security model

But one thing that TensorBoard doesn’t do is user authentication and authorization: there is no notion of user session, credentials, nor access control in TensorBoard, and no plan to implement any.

What it means in practice, is that TensorBoard is a great solution if you’re developing and testing things on your own laptop, where you’re the only user. But on shared environments like HPC clusters in general and Sherlock in particular, running a TensorBoard instance on a compute node means that any user on the cluster can connect to it, and interact with it, as if they were you: TensorBoard runs under your account, and through its unprotected web interface, it exposes your files and processes to any user that can connect to its web interface. There is no authentication mechanism.

Which is, for lack of better term, "not great".

Your own private and secure TensorBoard, on demand

Because it would be a shame to not be able to use such a valuable tool on our clusters, we came up with a solution to let users on Sherlock run TensorBoard in a secure and private way, without adding any additional configuration or access burden.

The TensorBoard OnDemand app, which is accessible through the Sherlock OnDemand portal, implements an authenticating reverse proxy that ensures that only the user who started the session can access it.

In a nutshell, by setting a browser cookie in the OnDemand interactive app page, we can make sure that the authenticating reverse proxy we developed and that controls access to the TensorBoard web interface, only authorize requests that come from the user authenticated through the OnDemand web page.

Without that cookie, access to the TensorBoard web interface is denied. And if the cookie is ever lost, users can simply re-create it by visiting the “My Interactive Sessions” page and clicking the “Connect” button again.

And if you’re curious about the details, or if you’re not using Sherlock and would like to implement a similar solution at your site, our TensorBoard OnDemand app is available on GitHub.

TL;DR

TensorBoard sessions on Sherlock are secure and private, in a completely transparent way.

It’s a little thing, but we hope it can make working on Sherlock more secure, without putting any additional configuration burden on the users.

So happy experimenting in TensorBoard, and as usual, please don’t hesitate to [email protected]" target="_blank" rel="noopener">reach out if you have any comments or questions.

Module versions get a facelift

2019-04-24T18:55:00.001Z

If you’re a Sherlock user who often needs to load specific module versions, but don’t care or can’t remember specific minor version numbers, you’re in luck!

Starting today, Sherlock allows loading modules via an extended default system which will automatically load the best available version, based on what users request.

Too many versions

There are too many version numbers to remember. For instance, take CUDA:

$ ml av cuda
--------------- devel -- compilers, MPI, languages, libs ---------------
cuda/8.0.61 (g) cuda/9.2.148 (g)
cuda/9.0.176 (g,D) cuda/10.0.130 (g)
cuda/9.1.85 (g) cuda/10.1.105 (g)
cuda/9.2.88 (g)
Where:
g: GPU support
D: Default Module

There are a number of versions of the CUDA module, that follow the semantic versioning scheme of major.minor.patch version numbers.

This is great, but in practice, most of us only care about the major version number. Sometimes major.minor, but mostly, we just want to get the latest patch version. Without having to look it up, let alone remember it.

Extended defaults

Enter extended defaults, a new module mechanism to simplify all of this.

Starting now, you can simply load a module using its major version number, and you’ll get the best module that matches your request.

For instance:

$ ml cuda/10.1

will automatically load cuda/10.1.105

If you want to specific a minor version, but don’t care about the patch number, you can too:

$ ml cuda/9.2

will load cuda/9.2.148

Finally, if you don’t specify any version at all, the highest version number will automatically be loaded, unless a default is defined (noticeable with the D flag in ml av output) which will be used instead.

We hope this will make things easier, and if you have any question, don’t hesitate to reach out at [email protected]" target="_blank" rel="noopener">[email protected]

Sherlock goes container-native

2018-11-13T22:31:00.001Z

In recognition of the fast-moving software landscape and the emerging needs of our users, we’re pleased to announce, from the SC18 Supercomputing conference in Dallas, TX, that Sherlock is going container-native.

Starting today, users won’t need to load any software module to run their Singularity or Docker containers on Sherlock. Singularity is now available natively on each and every login and compute node on Sherlock, and containers can be executed directly, no preliminary command required.

Oh, and just because we like to go all-in, we’re also throwing a brand new Singularity version into the mix. Singularity 3.0 brings many new features to the table, including:

base code re-write into Go
Singularity Image Format (SIF)
key management Platform (based in PGP) for signing and verification
integration With SyCloud services: Remote Build Service, Container Library Service, Key Management Service.

And more! For complete details, see https://www.sylabs.io/singularity.

To get started running your first containers on Sherlock, please check out our documentation about how to use Singularity on Sherlock at https://www.sherlock.stanford.edu/docs/software/using/singularity/

Running containers on Sherlock

2018-10-09T23:34:00.001Z

Have you ever wanted to run Docker containers on Sherlock, and skip the hassle of figuring out your application dependencies, installing them, compiling the code, resolving compilation errors, finding missing libraries, or making sure your application was performing optimally?

Well, thanks to Singularity, a container system especially designed to replace Docker on HPC systems, you can!

Singularity has been available on Sherlock since its early versions, but we just added new documentation that will show you how to use Singularity on Sherlock, and provide more examples of what you can do with containers.

You’ll find information on how to run Docker containers with Singularity, how to download and run GPU-accelerated applications from the NVIDIA GPU Cloud registry, and more.

Head over to https://www.sherlock.stanford.edu/docs/software/using/singularity for more details, and as usual, feel free to reach out to [email protected]" target="_blank">[email protected] if you have any question!

Sherlock now welcomes lab-contributed software

2018-07-25T01:11:00.001Z

Have you ever spent hours compiling and tuning software, put together a complex workflow combining multiple software parts, or just installed the latest version of the brand new software tools you're developing on Sherlock? And wondered how to share your results with the rest of the Sherlock community, so more people could benefit from your work?

Sherlock now offers the possibility for labs to share their software installations and modules, and to make them available to each and every user on the cluster.

A new labs section has been introduced in the software modules categories (complete details are available in the documentation).

Kudos to the Poldrack Lab, which has been pioneering this approach on Sherlock and offered to share their software modules with everyone.

This is a great way to encourage sharing, spark innovation, and improve user experience on Sherlock. So if you're interested and would like to share your software installations too, please [email protected]" target="_blank" rel="noopener">let us know and we'll be in touch!

MySQL on Sherlock

2018-07-20T21:36:00.001Z

Have you ever wondered how you could use a database server on Sherlock? Or wanted to run jobs that query data from a SQL database? Or even better, multiple jobs that would interact with the same database? Have you ever wanted to store datasets in a SQL format and be able to query them over long periods of time, from many different jobs?

Well, now you can.

Sherlock provides a installation of MariaDB, a free MySQL alternative, that you can run in the context of a job, and load with the following command:

$ ml system mariadb

You can submit a job that starts a MariaDB instance, and query it from within that job. You can also start a DB-server job, and then interact with it from other jobs, over the network.

And if you store the database data on one of the shared, persistent Sherlock filesystems, you'll find your database data back the next time you submit a DB-server job and start a MariaDB instance, without having to re-import anything.

For more information about Sherlock's self-service database-in-a-box option , see the documentation at:
https://www.sherlock.stanford.edu/docs/software/using/mariadb/

Spark on Sherlock

2018-05-16T20:43:00.001Z

Apache Spark is now available on Sherlock, via the spark module:

$ ml spark

We also added some new documentation and example job scripts, so if you ever needed to run Spark jobs on Sherlock, please take a look at those and [email protected]" target="_blank" rel="noopener">let us know how it works for you.

Tensorflow 1.7.0 now available

2018-03-30T18:15:00.001Z

Tensorflow 1.7.0, which features TensorRT integration, is now available on Sherlock 2.0.

You can load it on Sherlock with:

$ ml py-tensorflow/1.7.0_py27

NB: As with all the other Tensorflow modules on Sherlock, Tensorflow 1.7.0 requires a GPU to run and can only be used in jobs submitted to GPU nodes.

For a complete changelog, please see the TensorFlow release notes.