Pi _ Jupyter - dwisianto/dwisianto GitHub Wiki

  • ipykernel
  • ipynb
  • nbmake
  • nbsphinx

Jupyter in a Conda Environment

KernelSpec3

$ conda activate ml
(ml) $ conda install ipykernel
(ml) $ ipython kernel install --user --name=<any_name_for_kernel>
(ml) $ conda deactivate
  • How To Install Jupyter Notebook And The Kernel In Your Conda Environmenthow-to-install-jupyter-notebook-and-the-kernel-in-your-conda-environment
  • Assuming your conda-env is named ml, it is as simple as:
  • conda install ipykernel installs all dependencies needed to use jupyter.
  • ipython kernel install --user --name=<any_name_for_kernel> installs the kernel for this environment. I usually use the same kernel name as the environment name here (i.e. ml in this example).
  • Run jupyter kernelspec list to get the paths of all your kernels.
  • jupyter kernelspec list
  • Then simply uninstall your unwanted-kernel:
  • jupyter kernelspec uninstall unwanted-kernel

Kernelspec1

https://queirozf.com/entries/jupyter-kernels-how-to-add-change-remove

  • jupyter kernelspec list
  • jupyter kernelspec add
  • jupyter kernelspec remove
  • python -m ipykernel install --user --name mycustomkernel

Kernelspec2

How To Install Jupyter Notebook And The Kernel In Your Conda Environmentlink

Assuming your conda-env is named ml, it is as simple as:

$ conda activate ml
(ml) $ conda install ipykernel
(ml) $ ipython kernel install --user --name=<any_name_for_kernel>
(ml) $ conda deactivate

conda install ipykernel installs all dependencies needed to use jupyter.

ipython kernel install --user --name=<any_name_for_kernel> installs the kernel for this environment. I usually use the same kernel name as the environment name here (i.e. ml in this example).

Machine One

  • Location
    • cd d2/s2/c2/pj
    • jupy.sh
    • conda activate jupy39
    • jupy.sh
#
# JUPY
#
export JUPY_HOME=/anaconda3/envs/jupy39/
export JUPY_WORK=~/d/s/m/wk/
alias jpl="cd $JUPY_WORK; $JUPY_HOME/bin/jupyter-lab"
echo "   JUPY: JUPY_HOME $JUPY_HOME"
echo "         JUPY_WORK $JUPY_WORK"

Convert Notebook to python script

one file

which jupyter-nbconvert
jupyter nbconvert --to python my_notebook.ipynb

all notebooks in the directory

pyt_exe="/mlstorage/mansjurd/d21/apps/ana/c202106a/envs/og21a/bin/pytest"
jupy_exe="/mlstorage/mansjurd/d21/apps/ana/c202106a/envs/og21a/bin/jupyter"
work_dir=""


echo "find .  -name '*.ipynb' ! -name '*checkpoint.ipynb' -exec ${jupy_exe} nbconvert --to python {} \;"
find .  -name '*.ipynb' ! -name '*checkpoint.ipynb' -exec ${jupy_exe} nbconvert --to python {} \;
pytest

Logging

One

"""
In jupyter notebook simple logging to console
"""
import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)

# Test
logger = logging.getLogger('LOGGER_NAME')

logger.debug('This is hidden')
logger.info('So this is shown on the console')
logger.warning('This too')

Two

"""
In jupyter notebook simple logging to console and file:
"""
import logging
import sys

logging.basicConfig(
    level=logging.INFO, 
    format='[{%(filename)s:%(lineno)d} %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler(filename='tmp5a.log'),
        logging.StreamHandler(sys.stdout)
    ]
)

# Test
logger = logging.getLogger('LOGGER_NAME')
logger.debug('This message should go to the log file and to the console')
logger.info('So should this')
logger.warning('And this, too')

Logging Three

"""
Setup simple logging in python. This logs info message to stdout and debug messages to file.
Sure it's long but this is as simple as I could make it for this outcome.
Note: We must set the root logger at DEBUG level, since it must be higher than it's children to pass them on.
Then set filehandler at debug and stream handler at info.
"""
import logging
import sys
import datetime

# To use differen't log level for file and console
timestamp = datetime.datetime.utcnow().strftime('%Y%m%d_%H-%M-%S')
filename=f'/tmp/tmp5a_{timestamp}.log'
formatter = logging.Formatter('[%(asctime)s] %(name)s {%(filename)s:%(lineno)d} %(levelname)s - %(message)s')

file_handler = logging.FileHandler(filename=filename)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)

stream_handler = logging.StreamHandler(sys.stdout)
stream_handler.setLevel(logging.INFO)

# The handlers have to be at a root level since they are the final output
logging.basicConfig(
    level=logging.DEBUG, 
    format='[{%(filename)s:%(lineno)d} %(levelname)s - %(message)s',
    handlers=[
        file_handler,
        stream_handler
    ]
)

# Test
logger = logging.getLogger('LOGGER_NAME')
logger.debug('This message should go to the log file')
logger.info('This should go to the stdout and file')
logger.warning('And this, too')
print('filelog', open(filename).read())