Fixed wrong "chrom" column dtype inference in pandas

[lillux]

Hello all,

tanks for developing PhyClone to perform phylogenetic inference from PyClone-VI pre-clustering.

I was testing the tool on some big WGS data from multiple tumor samples from mice, with input .tsv with some millions rows, and I guess I got stuck on an old pandas type inference bug, that was inferring wrong data type for the chrom column, that in my case was composed mostly of integer (chroms 1 to 19) and some clear strings ("X" and "Y").

Error:

phyclone/data/validator/input_validator.py:87: DtypeWarning: Columns (0: chrom) have mixed types. Specify dtype option on import or set low_memory=False. return pd.read_csv(file_path, sep=csv_delim)
InputFormatError: Input format error(s) found in file: phyclone/phyclone_input/25_mouse_phyclone_input.tsv Invalid column(s), affected column count: 1 - chrom: Column contains elements that violate the required minimum string length of 1

The error was not happening with the control samples (few mutations in the order of 10K rows). I also tested a simple pd.read_table() and the same DtypeWarning was popping up with the big .tsv but not with the small one, suggesting pandas responsibility in the bug.

I manually checked the inputs for wrongly formatted chromosome names and presence of nan, but none of the failing input showed problems other than wrong pandas dtype inference.

Here I propose a patch that adds a small wrapper around read_csv() function, explicitly defining the string dtype to the chrom columns, while keeping existing sanity checks unaltered.

The patch allow me to run the inference, that is still ongoing.

I'm always available if additional tests are required.

Best,
Lillo

[elhurtado]

Hello Lillo,

Thank you for bringing this to my attention and taking the time to put together this PR, it is much appreciated! You've definitely found an edge case I hadn't encountered or accounted for.

While the proposed fix in this PR is generally quite good, when digging into the underlying cause a bit (thank you for the stackoverflow link as well) I found that this same form of edge case could be triggered in other scenarios that wouldn't be covered within the current PR; e.g. if the mutation_id, sample_id, or any of the other columns were to also have ambiguously parsed column data types. As a second rather minor thing, this proposed fix would also necessitate me having to redefine the data and cluster schemas, which while I ultimately may decide to do further down the line, it isn't currently planned.

Due to the above I have implemented my own fix following a different strategy, it can be found in this branch: https://github.com/Roth-Lab/PhyClone/tree/input_validator_pandas_dtype

If you're willing, feel free to try out that branch and see if it addresses the issue for your dataset. If you encounter any problems with it, feel free to let me know.

Either way, thank you again for having found this bug and pointing me in the right direction for the solution.

Best,
Emilia