Python_Notes.md

Virtual Environment

Conda

• Create Env

tags: # conda create env # create env # conda create virtualenv

Example to create an conda env named my_env :

conda create --name my_env python=3.10

Activate:

conda activate my_env

Note

Recommend to upgrade pip, setuptools, wheel when first time established environment

pip install --upgrade pip setuptools wheel

Pip

• limit range of package version

tag # pip limit version # pip version smaller than # package version smaller # package smaller version

eg

pip install "numpy<2"

• search packages, filter packages

tag # pip filer # pip search packages # pip filter out packages # pip find # pip find certain # pip find by string # pip search packages with string

# linux
pip list | grep tensorflow

# windows
pip list | findstr tensorflow

Colab

• run py file with multiple input arguments, by adding %%bash on top of cell¹:

tag # colab multiple arg # multiple argv # colab wrap command # colab cell multiple parameters

%%bash
python example.py arg0=0 \
                    arg1=1 \
                    arg2=2 \

$\quad$ Note that adding # is not allowed in bash cell, no space after \

• Edit and overwrite files in colab

tag # colab edit files # colab edit py # colab edit and save # colab modify files

in cell, run command %pycat *filename*, like:

%pycat utils.py

Will appear a pop up, shows codes

Copy and paste contents into new empty cell like:

# this is a cell in colab
# contents of utils.py
import numpy as np

def show():
    print()

# something
# something
#

Edit

# this is a cell in colab
# contents of utils.py
import numpy as np

def show():
    # some modified
    print("show modified")
    print()

# something
# something
#

After edit finished, add command %%writefile *filename* on top of the cell

%%writefile utils.py
# this is a cell in colab
# contents of utils.py
import numpy as np

def show():
    # some modified
    print("show modified")
    print()

# something
# something
#

Finally, run the cell, file will be modified

Reference

• Update code in colab after file changed²

tag # colab autoreload # colab update code # autoreload # colab update files # colab file changed

import sys
import importlib

# withdraw file extension .py
importlib.reload(sys.modules["utils"])
importlib.reload(sys.modules["tools.data_loader"])

Then import functions again

from utils import show
from tools.data_loader import test_loader

• location of colab files

Usually at /content/, able to access from root folder

Hydra

tag #hydra #hyper para #hyper parameter

Here's an example³:

• set sub-config file as parameters for py file, in folder conf/

defaults:
  - db: mysql

debug: false

$\quad$ Means that there's a yaml file in folder conf/db/

$\quad$ Prepare two yaml files in conf/db:

# mysql.yaml
name: mysql

#sqlite.yaml
name: sqlite

$\quad$ Prepare test py file on root folder:

import hydra
from omegaconf import DictConfig, OmegaConf

@hydra.main(version_base=None, config_path="conf", config_name="config")
def my_app(cfg : DictConfig) -> None:
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    my_app()

$\quad$ looks like this

├── my_app.py
├── conf
│   ├── config.yaml
│   ├── db
│   │   ├── mysql.yaml
│   │   └── sqlite.yaml 

• Run commend line

python my_app.py # call default db in config/config.yaml

python my_app.py db=sqlite # change db to sqlite, in config/config.yaml

References

Footnotes

1. from jakevdp stackoverflow ↩

2. from Antony Hatchkins stackoverflow ↩

3. hydra official example1 and example2 ↩