Sherlock changelog

Final hours announced for the June 2023 SRCF downtime

2023-05-12T22:30:44.259Z

As previously announced, the Stanford Research Computing Facility (SRCF), where Sherlock is hosted, will be powered off during the last week of June, in order to safely bring up power to the new SRCF2 datacenter.

Sherlock will not be available for login, to submit jobs or to access files from Saturday June 24th, 2023 at 00:00 PST to Monday July 3rd, 2023 at 18:00 PST.

Jobs will stop running and access to login nodes will be closed at 00:00 PST on Saturday, June 24th, to allow sufficient time for shutdown and pre-downtime maintenance tasks on the cluster, before the power actually goes out. If everything goes according to plan, and barring issues or delays with power availability, access will be restored on Monday, July 3rd at 18:00 PST.

We will use this opportunity to perform necessary maintenance operations on Sherlock that can’t be done while jobs are running, which will avoid having to schedule a whole separate downtime. Sherlock will go offline in advance of the actual electrical shutdown to ensure that all equipment is properly powered off and minimize the risks of disruption and failures when power is restored.

A reservation will be set in the scheduler for the duration of the downtime: if you submit a job on Sherlock and the time you request exceeds the time remaining until the start of the downtime, your job will be queued until the maintenance is over, and the squeue command will report a status of ReqNodeNotAvailable (“Required Node Not Available”).

The hours leading up to a downtime are an excellent time to submit shorter, smaller jobs that can complete before the maintenance begins: as the queues drain there will be many nodes available, and your wait time may be shorter than usual.

As previously mentioned, in anticipation of this week-long downtime, we encourage all users to plan their work accordingly, and ensure that they have contingency plans in place for their computing and data accessibility needs during that time. If you have important data that you need to be able to access while Sherlock is down, we strongly recommend that you start transferring your data to off-site storage systems ahead of time, to avoid last-minute complications. Similarly, if you have deadlines around the time of the shutdown that require computation results, make sure to anticipate those and submit your jobs to the scheduler as early as possible.

We understand that this shutdown will have a significant impact for users who rely on Sherlock for their computing and data processing needs, and we appreciate your cooperation and understanding as we work to improve our Research Computing infrastructure.

For help transferring data, any questions or concerns, please do not hesitate to reach out to [email protected]" rel="noopener nofollow" target="_blank">[email protected].

SRCF is expanding

2023-02-24T02:00:00Z

The Stanford Research Computing Facility (SRCF), where Sherlock is hosted, has been a highly effective data center since its opening in January of 2014, and demand has grown so much that we’re expanding it! Another identical building (SRCF2) is under construction at SLAC, which will increase our data center capacity when it opens this summer.

In order to bring power to the new building, the entire existing SRCF data center will need to be shut down. The 12kV electrical infrastructure is so pervasive that for the new building to be connected safely, everything needs to be powered off, including the backup generators. It unfortunately means that all servers and equipment will need to be shut down for this event, including Sherlock.

The full building power shutdown is planned for late June 2023, it’s expected to last for about a week, and Sherlock will be unavailable during that time.

During the power outage, Sherlock will be entirely powered down, meaning that it will not allow login or data transfer, the Sherlock OnDemand interface will be down, jobs will not run, and data will not be accessible (including $HOME, $SCRATCH and $OAK). We expect all services to resume normally once power is back up, and jobs that were in queue before the downtime should resume being scheduled normally.

The power outage is currently scheduled for the last week of June 2023. Specific dates and times have not been finalized yet, but we will share more detailed information as the shutdown date gets closer.

In anticipation of this week-long downtime, we encourage all users to plan their work accordingly, and ensure that they have contingency plans in place for their computing and data accessibility needs during that time. If you have important data that you need to be able to access while Sherlock is down, we strongly recommend that you start transferring your data to off-site storage systems ahead of time, to avoid last-minute complications. Similarly, if you have deadlines around the time of the shutdown that require computation results, make sure to anticipate those and submit your jobs to the scheduler as early as possible.

We understand that this shutdown will have a significant impact for users who rely on Sherlock for their computing and data processing needs, and we appreciate your cooperation and understanding as we work to improve our Research Computing infrastructure.

For help in transferring data, any questions or concerns, please do not hesitate to reach out to [email protected]" rel="noopener nofollow" target="_blank">[email protected].

Next scheduled maintenance: Oct. 16

2019-10-14T21:23:00.001Z

In order to prepare future improvements of the parallel /scratch file system, as well as performing some required work on the scheduler, Sherlock will not be available during the following times:

Wednesday, October 16th, 2019 - from 8:00am to 6:00pm

Access to Sherlock will be unavailable, logins will be disabled and jobs won’t run during that period.

Note: if you submit a job on Sherlock before the downtime and the time you request exceeds the time remaining until the maintenance begins, your job will run when the maintenance is over. The hours leading up to the maintenance are often a good time to submit shorter, smaller jobs that can complete before the maintenance begins: as the queues drain there will be many nodes available, and your wait time may be shorter than usual.

Oak storage will stay online during Sherlock’s maintenance, including all its gateways. If you need to access files on Oak during that time, you can use Globus, the shared DTN (eg. SSHFS), or other private NFS/SMB Oak gateways if you have any.

Next scheduled maintenance: Feb. 5

2019-01-23T00:17:00.001Z

Sherlock will not be available on Tuesday, February 5th, 2019 from 8:00am to 6:00pm.

System maintenance and upgrades will be performed during this time. Access to Sherlock will be unavailable, logins will be disabled and jobs won’t run.

If you submit a job on Sherlock and the time you request exceeds the time remaining until the maintenance begins, your job will run when the maintenance is over.

Note that the hours leading up to the maintenance are an excellent time to submit shorter, smaller jobs that can complete before the maintenance begins: as the queues drain there will be many nodes available, and your wait time may be shorter than usual.

Next scheduled maintenance: Nov. 28

2018-11-14T19:33:00.001Z

Sherlock will not be available on Wednesday, November 28th, 2018 from 8:00am to 6:00pm.

System maintenance and upgrades will be performed during this time. Access to Sherlock will be unavailable, logins will be disabled and jobs won’t run.

If you submit a job on Sherlock and the time you request exceeds the time remaining until the maintenance begins, your job will run when the maintenance is over.
The squeue command will report a pending reason of ReqNodeNotAvailable (“Required Node Not Available”) for those jobs.

Note that the hours leading up to the maintenance are an excellent time
to submit shorter, smaller jobs that can complete before the maintenance
begins: as the queues drain there will be many nodes available, and your
wait time may be shorter than usual.

Hours announced for the SRCF shutdown

2018-08-13T15:40:00.001Z

As previously announced, the Stanford Research Computing Facility (SRCF) will undergo major power maintenance over the Labor Day week-end (Sep. 1-3, 2018).

Because it’s hosted in that datacenter, Sherlock will not be available for login, to submit jobs or to access files from Friday, Aug. 31ˢᵗ at noon to Tuesday, Sep. 4ᵗʰ at 9am.

Jobs will stop running and access to login nodes will be closed at noon on Friday, Aug 31st, to allow sufficient time for shutdown and pre-downtime maintenance tasks on the cluster, before the power actually goes out. If everything goes according to plan, access will be restored on Tuesday, Sep. 4 at 9am.

A reservation will be set in the scheduler for the duration of the downtime: if you submit a job on Sherlock and the time you request exceeds the time remaining until the start of the downtime, your job will be queued until the maintenance is over, and the squeue command will report a status of ReqNodeNotAvailable (“Required Node Not Available”).

Note that the hours leading up to a downtime are an excellent time to submit shorter, smaller jobs that can complete before the maintenance begins: as the queues drain there will be many nodes available, and your wait time may be shorter than usual.

SRCF Reboot

2018-05-31T23:57:00.001Z

A shutdown of the Stanford Research Computing Facility (SRCF) is planned for Labor Day weekend, Saturday through Monday, September 1-3, due to the need for power system maintenance.

We will bring Sherlock down on Friday evening, August 31, and expect to bring it back up on Monday evening, September 3, 2018.

During that time, jobs will not run, file systems will not be available, and login nodes will be offline. All network connectivity to Sherlock will be interrupted.

Sherlock will not be available for login, to submit jobs or to access files.

We will place a reservation in the scheduler, so jobs won't be allowed to start if they don't have a chance to end before the downtime.

We know that this is a disruptive event, but power maintenance is a necessary operation, and all servers and equipment at the SRCF will need to be shut down, including the generators, when the power goes off. The Labor Day weekend was chosen for this event because it has consistently lower power use, implying lowest load at the SRCF.

Please see the SRCF reboot page for additional details, and don't hesitate to [email protected]" target="_blank" rel="noopener">reach out if you have any question.

Next scheduled maintenance: Jan. 31

2018-01-18T00:18:02Z

Sherlock 2.0 will not be available on Wednesday, January 31st, 2018 from 8:00am to 2:00pm

System maintenance and upgrades will be performed during this time. Access to Sherlock 2.0 will be unavailable, logins will be disabled and jobs won't run during that period.

Sherlock 1.0 will remain available during that time.

If you submit a job on Sherlock 2.0 and the time you request exceeds the time remaining until the maintenance begins, your job will run when the maintenance is over. The squeue command will report "ReqNodeNotAvailable" ("Required Node Not Available").

Next scheduled maintenance: Dec. 12

2017-11-28T19:30:34Z

Sherlock 2.0 will not be available on Tuesday, December 12th, 2017 from 8:00am to 2:00pm

System maintenance and upgrades will be performed during this time. Access to Sherlock 2.0 will be unavailable, logins will be disabled and jobs won't run during that period. Sherlock 1.0 will remain available during that time.

Next scheduled maintenance: Sept. 20

2017-09-06T23:59:08Z

Sherlock will not be available on Wednesday, September 20 2017 between 9:00 am and 2:00 pm.

System maintenance and upgrades will be performed during this time. Access to Sherlock will be unavailable, logins will be disabled and jobs won't run during that period.

We placed a maintenance reservation in the scheduler, so no new job will be allowed to start if it doesn't have a chance to end before the downtime.