Taxoble

Taxoble is an automated framework for constructing taxonomies for a domain from a given term set.

The pipeline can be adapted to any domain with a list of candidate terms or article keywords.

Repository Layout

.
├── construct_taxonomy.ipynb          # Main notebook for taxonomy construction
├── download_data_from_pubmed.ipynb   # Optional PubMed data download notebook
├── preprocess_input.py               # Candidate term preprocessing script
├── requirements.txt                  
├── LICENSE
├── .gitignore
└── taxonomy/
    ├── __init__.py                   
    ├── config.py                     # Pipeline configuration dataclasses
    ├── llm_client.py                 # Ollama/OpenAI adapter
    ├── embeddings.py                 # Sentence-transformer embedding helpers
    ├── nmf_pipeline.py               # NMF topic grouping utilities
    ├── llm_taxonomy.py               # LLM prompts for taxonomy construction
    ├── construction_pipeline.py      # Fine and broad taxonomy passes
    ├── graph_utils.py                # Hierarchy and DAG cleanup helpers
    ├── cycle_utils.py                # Cycle detection and repair
    ├── merge_utils.py                # Similar-node merging
    └── io_utils.py                   # IO utils

Workflow

1. Optional: Download PubMed Data

Use download_data_from_pubmed.ipynb to fetch PubMed records for selected journals and year ranges. The notebook saves CSV files under data/raw/.

You can skip this step if you already have a CSV file with candidate keywords.

2. Preprocess Candidate Terms

Edit the configuration block at the bottom of preprocess_input.py for your domain, input file, and keyword column.

Then run:

python preprocess_input.py

This creates keyword_mapping.pkl and supporting cache files in data/processed/<folder_name>/.

Preprocessing modes:

• canonicalize: expands abbreviations and merges strict synonyms.
• canonicalize_and_filter_by_domain: also removes terms that are not relevant to the configured domain.

3. Construct the Taxonomy

Open construct_taxonomy.ipynb and edit the TaxonomyConfig cell.

Run the notebook cells in order. The notebook performs:

1. Candidate term loading from keyword_mapping.pkl.
2. Fine-grained taxonomy construction.
3. One or more broad taxonomy construction passes.
4. Final term (keyword)-node (topic) mapping and taxonomy export.

Outputs

Intermediate artifacts are saved under:

data/processed/<folder_name>/<experiment_name>/

The readable final hierarchy is written to:

results/<folder_name>/<experiment_name>/final_taxonomy.txt

Edges are represented as (child, parent) tuples.

Notes

• LLM calls can be slow and may incur API costs if using a hosted provider.
• The pipeline relies on LLM. Review final outputs before using them in downstream analyses.
• Some cycle resolution may require manual input when the LLM cannot confidently repair a graph; in this case, an interactive prompt will appear and wait for the user to choose which edge to remove.
• Large keyword sets can require substantial memory for embedding and similarity computations.

Citation

If you use this code in a publication or shared project, please cite the repository.