Sherlock changelog

www.sherlock.stanford.edu

A new tool to help optimize job resource requirements

by Kilian Cavalotti, Technical Lead & Architect, HPC

It’s not always easy to determine the right amount of resources to request for a computing job. Making sure that the application will have enough resources to run properly, but avoiding over-requests that would make the jobs spend too much

Tracking NFS problems down to the SFP level

by Kilian Cavalotti

This is part of our technical blog series about things that happen behind-the-scenes on Sherlock, and which are part of our ongoing effort to keep it up and running in the best possible conditions for our beloved users. For quite a long

Tracking NFS problems down to the SFP level

Sherlock is hard at work against COVID-19

by Kilian Cavalotti,

About a month ago, we announced that we were dedicating a portion of Sherlock's computing resources to research projects around COVID-19. Since then, more than 15 PIs and research groups have reached out to share their projects, and their

Adventures in storage

by Kilian Cavalotti,

_This is part of our blog series about behind-the-scenes things we do on a regular basis on Sherlock, to keep it up and running in the best possible conditions for our users. Now that Sherlock's old storage system has been retired, we can

When setting an environment variable gives you a 40x speedup

by Kilian Cavalotti,

Today, we’d like to share some of our recent work on Sherlock that allowed a pretty significant speedup when listing files in directories with a lot of entries. Unlike our usual announcements, this post is more of a behind-the-scenes

When setting an environment variable gives you a 40x speedup

Persistent processes on Sherlock

by Kilian Cavalotti,

There are many cases where having a persistent process (or service) running alongside computing jobs is of great benefit. For instance, when data is stored in a database format to allow for highly efficient queries, jobs that want to

Running containers on Sherlock

by Kilian Cavalotti,

Have you ever wanted to run Docker containers on Sherlock, and skip the hassle of figuring out your application dependencies, installing them, compiling the code, resolving compilation errors, finding missing libraries, or making sure your

Running containers on Sherlock

MySQL on Sherlock

by Kilian Cavalotti,

Have you ever wondered how you could use a database server on Sherlock? Or wanted to run jobs that query data from a SQL database? Or even better, multiple jobs that would interact with the same database? Have you ever wanted to store

Spark on Sherlock

by Kilian Cavalotti,

Apache Spark is now available on Sherlock, via the spark module: $ ml spark We also added some new documentation and example job scripts, so if you ever needed to run Spark jobs on Sherlock, please take a look at those and let us know

Newer