Jupyter - problems with dotnet-try

Since .NET Try supports F# officially I wanted to switch over my Jupyter notebook instance to use that instead of IFsharp. But I ran into a couple of frustrating issues that I wanted to document in case anyone else hit them and didn't know how to debug the issue.

Issue 1. dotnet-try installs but isn't usable

I followed Scott Hanselman's instructions to install dotnet-try, but when I tried to execute dotnet try jupyter install it seemed as though dotnet-try wasn't installed at all:

$ dotnet tool install dotnet-try --global
You can invoke the tool using the following command: dotnet-try
Tool 'dotnet-try' (version '1.0.19553.4') was successfully installed.

$ dotnet try jupyter install Could not execute because the specified command or file was not found. Possible reasons for this include: * You misspelled a built-in dotnet command. * You intended to execute a .NET Core program, but dotnet-try does not exist. * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.

After a lot of head scratching I dug into the actual docs and learned that both .NET Core 3.0 and .NET Core 2.1 SDKs should be installed, while I only had .NET Core 3.1's SDK So after a quick sudo apt install dotnet-sdk-3.0 dotnet-sdk-2.1 I was successfully able to install the kernel and list it:

$ jupyter kernelspec list
Available kernels:
  .net-csharp    /home/notebook/.local/share/jupyter/kernels/.net-csharp
  .net-fsharp    /home/notebook/.local/share/jupyter/kernels/.net-fsharp
  mit-scheme     /usr/local/share/jupyter/kernels/mit-scheme
  python3        /usr/local/share/jupyter/kernels/python3

Issue 2. Jupyter can't run dotnet-try

However even though it was installed, each time I tried to create a new F# notebook Jupyter would give an error saying that it was unable to connect to the kernel. After taking a quick look at my logs I saw the same error as before!

Feb 09 13:19:11 aviemore jupyter[837]: [I 13:19:11.169 NotebookApp] KernelRestarter: restarting kernel (1/5), new random ports
Feb 09 13:19:11 aviemore jupyter[837]: Could not execute because the specified command or file was not found.
Feb 09 13:19:11 aviemore jupyter[837]: Possible reasons for this include:
Feb 09 13:19:11 aviemore jupyter[837]: * You misspelled a built-in dotnet command.
Feb 09 13:19:11 aviemore jupyter[837]: * You intended to execute a .NET Core program, but dotnet-try does not exist.
Feb 09 13:19:11 aviemore jupyter[837]: * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.
Feb 09 13:19:14 aviemore jupyter[837]: [I 13:19:14.180 NotebookApp] KernelRestarter: restarting kernel (2/5), new random ports
Feb 09 13:19:14 aviemore jupyter[837]: Could not execute because the specified command or file was not found.
Feb 09 13:19:14 aviemore jupyter[837]: Possible reasons for this include:
Feb 09 13:19:14 aviemore jupyter[837]: * You misspelled a built-in dotnet command.
Feb 09 13:19:14 aviemore jupyter[837]: * You intended to execute a .NET Core program, but dotnet-try does not exist.
Feb 09 13:19:14 aviemore jupyter[837]: * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH. 
I restarted the service, the server, checked and it took a while to realise the root cause. What happened was, even though dotnet-try was in PATH when I switched over to my jupyter use, it wasn't the case when I ran it via systemd. It seems that /etc/profile - which runs all the scripts in /etc/profile.d, one of which adds the ~/.dotnet/tools dir to PATH - is not used when systemd starts the service. I don't know the correct way to address this, but I wrote a quick script to setup PATH correctly and added an EnvironmentFile to the [Service] section my notebook.service file to use it:
EnvironmentFile=/home/jupyter/config/notebook-config
After I set this up and restarted notebook.service it was able to access dotnet-try and spin up the F# kernel correctly.

Fedora - Jupyter on a Linux remote using systemd

When you want to do some experimentation or put together a simple code-based presentation Jupyter notebooks are a powerful tool to have at your disposal. But if you use a number of devices over a few locations it can be useful to have a single instance hosted somewhere central (Linode, Digital Ocean, wherever) that you can access from any device wherever you are. There are a handful of ways that you can achieve this:

  1. log in to your remote machine, set Jupyter up and run jupyter notebook (perhaps in a tmux session) then log out - do this whenever your machine reboots
  2. as above but using an existing docker image
  3. spin up an Azure notebook
  4. ... or we could do something like #1 - but have it setup under a separate user and administered via a systemd service

All four of the above are fine for different reasons and use-cases but here I'll talk about how I put #4 together in a little Linode instance running Fedora 25 - it's relatively simple, you can control over the kernels installed, and it's another excuse to get a bit more in-depth with another Linux subsystem (systemd).

Requirements

All you need is a Linux system which uses systemd (Fedora 15.0 or newer, Debian 8.x or newer, Ubuntu 15.04 or newer, for example) which you have sudoer level access on, and Python 3.x. It's probably pretty straight-forward to set this up on systems using the SysV init but I won't cover them here.

Install and Set Up Jupyter 

First thing we need to do is install Jupyter and set up the user context which the Jupyter will be run under - which is a user called "jupyter":

$ sudo python3 -m ensurepip
$ sudo pip install jupyter
$ sudo useradd jupyter
$ sudo passwd jupyter

Next we should switch to the new jupyter user, create the directory our notebooks will live in and generate the Jupyter config we'll mess around with:

$ su - jupyter
$ mkdir notebooks
$ jupyter notebook --generate-config

The last command will create a new file ~/.jupyter/jupyter_notebook_config.py which we'll do a little messing around with shortly, but before this we'll set up a password 

$ python3
Python 3.5.2 (default, Sep 14 2016, 11:28:32) 
[GCC 6.2.1 20160901 (Red Hat 6.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from notebook.auth import passwd
>>> passwd() # below I enter "password123"
Enter password: 
Verify password: 
'sha1:2eff88aac285:385c87867bd18fe852ee1d56b1010d4beed96969'

This will be used to log in to the application when its running. Open up the ~/.jupyter/jupyter_notebook_config.py file in a text editor and add/modify the following lines (using the SHA1 hash returned by the above):

c.NotebookApp.port = 8888
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.password = 'sha1:2eff88aac285:385c87867bd18fe852ee1d56b1010d4beed96969'

Setting up a Jupyter systemd service

Now we want to create a new systemd service so we can make sure our Jupyter notebook runs on startup, handles logging nicely and has all the other bells-and-whistles afforded to us by systemd. This is surprisingly simple - we want to create a new file jupyter.service in /usr/lib/systemd/system - this will tie together our newly installed Jupyter software and our newly setup jupyter user - using your favourite text editor create it so it looks like the below:

$ sudo cat /usr/lib/systemd/system/jupyter.service
[Unit]
Description=Jupyter

[Service]
Type=simple
PIDFile=/var/run/jupyter.pid
ExecStart=/usr/bin/jupyter notebook --no-browser
WorkingDirectory=/home/jupyter/notebooks
User=jupyter
Group=jupyter

[Install]
WantedBy=multi-user.target%

Now all that's left to do is cross our fingers, enable our services, kick them off and browse to our remote box and login with our password:

$ sudo systemctl daemon-reload
$ sudo systemctl enable jupyter
$ sudo systemctl start jupyter

And if you want you can stop here - bookmark your http://www.xxx.yyy.zzz:port address and you're all set!

Conclusion

This was initially just an experiment - an excuse to test out my ability to put together a systemd .service file and do something more with a mostly-idle linux server sitting in a facility somewhere in Amsterdam. However I have found that I really like using this setup. When I was first shown Jupyter (née IPython) I was unimpressed and didn't see the point. However over the last few days I've been working through Project Euler problems again while teaching myself F# (using the IfSharp kernel) and I have found that it lends itself very well to my problem solving workflow on Project Euler.