科学计算环境 - duxiaoyao/pdp Wiki

Better practice

2021-05-12: 推荐使用pyenv + pipx + poetry,不再推荐conda
不用anaconda:带太多用不到的科学计算包,特别大
使用miniconda:只有Python和conda,没有预带的包,在用到时使用conda安装
不使用pyenv管理miniconda版本,可能有兼容问题,无法使用最新的miniconda,主流是直接使用conda
使用conda管理环境、包和依赖
参考设置镜像源设置.condarc
不建议使用pycharm anaconda定制版,推荐使用pycharm CE普通版

poetry

poetry new jl
cd jl
#!!! use aliyun pypi
poetry add pandas numexpr bottleneck pyarrow openpyxl numba cython scipy scikit-learn tabulate[widechars] matplotlib mplfinance seaborn plotly kaleido jupyterlab xeus-python ipywidgets ipympl statsmodels
poetry add typer[all] rich[jupyter] pydantic line_profiler pandas-profiling 
poetry add ta-lib gm quantstats 

常用工具

Helpers: conda/miniconda, ipython, jupyterlab
Analytics: Numpy, Scipy, Pandas
Visualization: ipywidgets(required by matplotlib & Plotly in jupyterlab), Matplotlib + ipympl(support Matplotlib in jupyterlab) + mplfinance + Seaborn, Plotly + Dash, tabulate[widechars]
Machine learning: Scikit-learn, PyTorch, PyCaret, jieba, NLTK, spaCy
Optimization: Numba, Cython / Pybind11 / Boost.Python, CuPy, NVIDIA CUDA / AMD ROC / Intel® Distribution for Python
Multiprocessing and parallization: Ray / Dask, modin, Vaex
Data store: feather V2+, parquet, CSV, Excel, pickle
Optional Packages/Tools: pandas-profiling, cufflinks(interactive pandas plots)

Install miniconda

  1. download Miniconda with Python 3.7
  2. install:
cd
bash Miniconda3-latest-Linux-x86_64.sh # input yes for miniconda initialization

pip config set global.index-url https://mirrors.aliyun.com/pypi/simple  
pip config set install.trusted-host mirrors.aliyun.com  

conda config --set auto_activate_base no
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
conda config --set ssl_verify yes
conda config --set show_channel_urls yes
source ~/.bashrc
  1. update:
conda info
conda update -n base conda
conda update --all
  1. prepare quant
conda install numpy scipy pandas numexpr bottleneck numba cython openpyxl scikit-learn statsmodels tabulate[widechars] matplotlib mplfinance seaborn plotly dash kaleido ipython xeus-python jupyterlab ipympl ipywidgets  
conda install ray # or conda install dask distributed fastparquet 
conda install typer[all] rich[jupyter] pydantic line_profiler pandas-profiling cufflinks
  1. prepare quant
conda create --clone base -n quant
conda env list
# install ta-lib (install on windows, check: https://blog.quantinsti.com/install-ta-lib-python/#windows)
cd ~/Downloads
wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz  # check on: https://ta-lib.org/hdr_dw.html
tar xzvf ta-lib-0.4.0-src.tar.gz
cd ta-lib
./configure --prefix=/usr
make
sudo make install
# install ta-lib python wrapper
cd
conda activate quant
pip install ta-lib
conda deactivate
  1. prepare gm
conda create --clone quant -n gm
conda activate gm
pip install gm
pip install tushare --upgrade
conda deactivate

Uninstall Miniconda

Conda usage

conda cheat sheet

conda info
conda list
conda config --describe auto_activate_base
conda update --all
conda search --full-name python
conda install python=3.7.3
conda info --envs
conda env list
conda create -h
conda create -n env1 python=3.7 numpy scipy pandas scikit-learn matplotlib
conda run -n env1 pip install TA-Lib
conda create --clone env1 -n env2
conda env remove -n env2
conda list -n env1
conda list -n env1 numpy
conda update --all -n env1
conda run -n env1 which python
conda activate env1
conda update -n base conda
conda update --all
conda update python=3.7.3
conda search/install/update/upgrade/remove/uninstall pandas
conda deactivate 
conda env remove -n env1