Added inter document masking for manual and flash attention. - #434

Open

BlueCrescent wants to merge 19 commits into

mainfrom

inter_document_masking_for_attention

Commits on Feb 23, 2026

feat(attention): Added inter document masking for manual and flash attention.
BlueCrescent
committed
5e039df
feat(data loading): GPT2LLMCollateFn can now determine the sub sequence lengths required for inter document attention masking.
BlueCrescent
committed
a5354c4

Commits on Feb 24, 2026

fix(attention): NaNs when using padding + inter document masking with manual attention.
BlueCrescent
committed
eba9c5b
feat(attention): added sub_seq_lengths_key to GPT2LLMConfig and renamed eod_token_id to eos_token_id
BlueCrescent
committed
3d583c1
fix(attention): added missing sub_seq_lengths_key parameter to get_gpt2_model
BlueCrescent
committed
cd00777
fix(attention): computing sub sequence lengths on correct input
BlueCrescent
committed
fdd6465
docs(attention): better _get_unpad_data_for_concatenated_sequences() doc string

BlueCrescent
and
Copilot
authored
e952bd0

Commits on Feb 25, 2026

test(attention): fixed collator tests and improved collator config validators
BlueCrescent
committed
115259c

Commits on Feb 26, 2026

fix: directly use correct device + dtype for eos positions extensions

BlueCrescent
and
Copilot
authored
8d71928

Commits on Feb 27, 2026

chore: remove comment

BlueCrescent
and
rrutmann
authored
91f67e4

Commits on Feb 28, 2026

Commits on Mar 12, 2026

chore: Merge remote-tracking branch 'origin/main' into inter_document_masking_for_attention
BlueCrescent
committed
b2ec756

Commits on Mar 30, 2026

chore: Merge remote-tracking branch 'origin/main' into inter_document_masking_for_attention
BlueCrescent
committed
956b958