If you’re into machine learning (and who isn’t these days?), you probably know all about TensorBoard already.

If you don’t, TensorBoard is TensorFlow's visualization toolkit. It provides the visualization and tooling needed for machine learning workflows: it enables tracking experiment metrics like loss and accuracy, visualizing model graphs, viewing histograms of weights, biases, or other tensors as they change over time, or profiling TensorFlow programs. All kinds of cool and useful stuff.

TensorBoard (lack of) security model

But one thing that TensorBoard doesn’t do is user authentication and authorization: there is no notion of user session, credentials, nor access control in TensorBoard, and no plan to implement any.

What it means in practice, is that TensorBoard is a great solution if you’re developing and testing things on your own laptop, where you’re the only user. But on shared environments like HPC clusters in general and Sherlock in particular, running a TensorBoard instance on a compute node means that any user on the cluster can connect to it, and interact with it, as if they were you: TensorBoard runs under your account, and through its unprotected web interface, it exposes your files and processes to any user that can connect to its web interface. There is no authentication mechanism.

Which is, for lack of better term, "not great".

Your own private and secure TensorBoard, on demand

Because it would be a shame to not be able to use such a valuable tool on our clusters, we came up with a solution to let users on Sherlock run TensorBoard in a secure and private way, without adding any additional configuration or access burden.

The TensorBoard OnDemand app, which is accessible through the Sherlock OnDemand portal, implements an authenticating reverse proxy that ensures that only the user who started the session can access it.

In a nutshell, by setting a browser cookie in the OnDemand interactive app page, we can make sure that the authenticating reverse proxy we developed and that controls access to the TensorBoard web interface, only authorize requests that come from the user authenticated through the OnDemand web page.

Without that cookie, access to the TensorBoard web interface is denied. And if the cookie is ever lost, users can simply re-create it by visiting the “My Interactive Sessions” page and clicking the “Connect” button again.

And if you’re curious about the details, or if you’re not using Sherlock and would like to implement a similar solution at your site, our TensorBoard OnDemand app is available on GitHub.

TL;DR

TensorBoard sessions on Sherlock are secure and private, in a completely transparent way.

It’s a little thing, but we hope it can make working on Sherlock more secure, without putting any additional configuration burden on the users.

So happy experimenting in TensorBoard, and as usual, please don’t hesitate to reach out if you have any comments or questions.