Errors with spades

[drpturner]

Apologies in advance if this is obvious..I am new!

I am using flowcraft via WSL2 (Ubuntu 20.04; singularity 2.5.2; i7 10th gen laptop with 16Gb RAM). Following the issues already noted, I downgraded nextflow to 18.10.1 and my pipeline comprising trimmomatic, fastqc, and skesa works fine. However, whatever I do, I cannot get spades to work (despite same version working fine outside of flowcraft): I get the following fail code every time.

Same issue when using a Windows workstation (same set-up, i7 7th gen with 64Gb RAM). Any suggestions on how to proceed...

(flowcraft) pault@POS-COMRU-NB04:/mnt/d/wgs/test2$ nextflow run pt2.nf --fastq "/mnt/d/wgs/fastq/_{1,2}."
N E X T F L O W ~ version 18.10.1
Launching pt2.nf [loving_woese] - revision: 44064f23ef

============================================================
F L O W C R A F T

Built using flowcraft v1.4.1

Input FastQ : 2
Input samples : 1
Reports are found in : ./reports
Results are found in : ./results
Profile : standard

Starting pipeline at Wed Sep 16 10:31:09 ICT 2020

[warm up] executor > local
WARN: Process configuration syntax $processName has been deprecated -- Replace process.$spades_1_2 = <value> with a process selector
[e6/ea457a] Submitted process > integrity_coverage_1_1 (29396_1#116)
[9f/746d95] Submitted process > status (29396_1#116)
[b5/3231e8] Submitted process > report_coverage_1_1
[97/6a649e] Submitted process > report (29396_1#116)
[a9/8acfda] Submitted process > spades_1_2 (29396_1#116)
[a9/8acfda] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (1)
[ba/e39263] Re-submitted process > spades_1_2 (29396_1#116)
[ba/e39263] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (2)
[77/68e9fe] Re-submitted process > spades_1_2 (29396_1#116)
[77/68e9fe] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (3)
[b7/5b334a] Re-submitted process > spades_1_2 (29396_1#116)
[b7/5b334a] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (4)
[6c/80b84f] Re-submitted process > spades_1_2 (29396_1#116)
[6c/80b84f] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (5)
[9b/e859a3] Re-submitted process > spades_1_2 (29396_1#116)
[9b/e859a3] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (6)
[6d/00578c] Re-submitted process > spades_1_2 (29396_1#116)
[6d/00578c] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (7)
[e3/029556] Re-submitted process > spades_1_2 (29396_1#116)
[e3/029556] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Error is ignored
[64/f0eabb] Submitted process > compile_status_buffer (1)
[7a/21accc] Submitted process > compile_reports (1)
[b2/1f8c8f] Submitted process > compile_status
Completed at: Wed Sep 16 10:33:04 ICT 2020
Duration : 1m 55s
Success : true
Exit status : 0

.nextflow.log
.command.log

[CreatorOfMoon]

I think i figured out why it does not work. at line 20 of your command log, when the file is running "spades.py" with the spades option, it does not run the actual Spades software, but it is running the Spades.py component of flowcraft again, with argument needed for spades software.

That's why Error log say there is not "SAMPLE_ID", of course cause the first argument is --careful and not sample id ^^'

The solution would be either changing the name of spades.py component of flowcraft for something else ( like fc_spades.py) and modify the pipeline accordingly,
Or changing this :
cli = [ "spades.py", "--careful", "--only-assembler", "--threads", "$task.cpus", "--cov-cutoff", min_coverage, "-o", "."

to this
cli = [ "$(which "spades.py") ", "--careful", "--only-assembler", "--threads", "$task.cpus", "--cov-cutoff", min_coverage, "-o", "."

in the spades.py part of flowcraft. Not sure about the second solution, but i can't figure out of an other way :P

[cimendes]

Hi! Sorry for not getting back to you on this issue! Unfortunately I haven't found the time yet to look deeply into your issue, but you seem to be on the right track with your solution! :) I invite you to do a PR if you want with your fix. If not, i'll try to correct it sometime this week.

Best,
Inês

[drpturner]

Many thanks for these replies...unfortunately no joy yet.

For the first option (changed template filename to fc_spades.py and updated the .nf file accordingly), spades still fails with error exist status (1). However the warnings / errors in the command.log are different:
== Warning == output dir is not empty! Please, clean output directory before run.
== Warning == No assembly mode was sepcified! If you intend to assemble high-coverage multi-cell/isolate data, use '--isolate' option.
== Error == system call for: "['/NGStools/SPAdes-3.14.0-Linux/bin/spades-core', '/tmp/nxf.Bx7cKswjiV/K21/configs/config.info', '/tmp/nxf.Bx7cKswjiV/K21/configs/careful_mode.info']" finished abnormally, OS return value: -11

I changed the fc_spades.py options to --isolate, but same result

If I try option (2), I get this error:
1 error
-- check template file: /mnt/d/wgs/p_spades/templates/fc_spades.py -- Execution is retried (1)
Unable to re-submit task spades_1_2 (29396_1#116)

[98/b63422] Submitted process > report (29396_1#116)
Workflow execution stopped with the following message:
Failed to parse template script (your template may contain an error or be trying to use expressions not currently supported): org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
/groovy/script/Script556B61ECD10F86B7E17B67558EBD5F7C: 201: illegal string body character after dollar sign;
solution: either escape a literal dollar sign "$5" or bracket the value expression "${5}" @ line 201, column 12.
"$(which "spades.py") ",
^
I made the change to "$(which "spades.py") " , but get the same message (with added ).

Thanks for any further thoughts.
Paul

Logs from option (1) test here:
.command.log
.nextflow.log

[CreatorOfMoon]

Hi drpturner, sorry to hear that ^^

I wasn't sure about the second option, sadly i don't think we can fix this problem by changing this line then.

For the first option, the problem seems to be Spades Software since this time it launched. Can you provide the log from Spades please ?
( params.txt and spades.log files from the output directory.

SPAdes log can be found here: /tmp/nxf.Bx7cKswjiV/spades.log )

[drpturner]

Thanks CreatorOfMoon - as far as I can tell that folder and log is not present in the tmp folder. Is it possible that this is part of the problem?

[cimendes]

Sorry, I haven't been hable to reproduced your error yet. I'm running the master version of flowcraft with nextflow (v 20.01.0.5264) and docker (v 19.03.12-ce). The spades.pt template is calling the spades.py executable in the container correctly, and I'm able to obtain an assembly. Could you please tell me what container software are you using?

[drpturner]

Hi - thanks for working on this. I'm using singularity v 2.5.2. It seems to be working ok and i can pull containers successfully.

…

________________________________ From: Inês Mendes <notifications@github.com> Sent: Wednesday, September 23, 2020 5:53:13 PM To: assemblerflow/flowcraft <flowcraft@noreply.github.com> Cc: Paul Turner <turner.drpaul@gmail.com>; Author <author@noreply.github.com> Subject: Re: [assemblerflow/flowcraft] Errors with spades (#233) Sorry, I haven't been hable to reproduced your error yet. I'm running the master version of flowcraft with nextflow (v 20.01.0.5264) and docker (v 19.03.12-ce). The spades.pt template is calling the spades.py executable in the container correctly, and I'm able to obtain an assembly. Could you please tell me what container software are you using? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#233 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMWZY7RQAJFFCSBZCAUC4WTSHHHRTANCNFSM4ROFADNA>.

[ramirma]

Could it be that you do not have write permissions to /temp with singularity?

[drpturner]

Thanks for this - I removed singularity and reinstalled (carefully) to version 3.6.3 following the instructions here: https://sylabs.io/guides/3.6/admin-guide/installation.html#installation-on-linux. There is a comment "By default Singularity will be installed in the /usr/local directory hierarchy."...could this be an issue? From the text that followed it seemed to imply that this would be ok re permissions, but not totally sure.

Is it likely that this would be isolated to spades...everything else I am using with flowcraft works fine - mostly a pipeline "trimmomatic fastqc skesa assembly_mapping pilon (prokka | mlst | abricate)". Runs on both machines with no errors.

Further tests showed this singularity update didn't change the outcome. I'm not sure what else to try - I might have to be happy with skesa and use spades outside of flowcraft (which works fine).

[drpturner]

An update - have just located the singularity.conf file and (I think) confirmed that /tmp is accessible:

MOUNT TMP: [BOOL]

DEFAULT: yes

Should we automatically bind mount /tmp and /var/tmp into the container? If

the --contain option is used, both tmp locations will be created in the

session directory or can be specified via the SINGULARITY_WORKDIR

environment variable (or the --workingdir command line option).

mount tmp = yes

[drpturner]

OK - progress. I changed the pipeline params.config so that the scratch option was false (disableRR_1_2 = false)...this meant the spades files all ended up in the pipeline /work folder. Spades still fails - log file attached if anyone has any remaining energy to take a look!

spades.log

[cimendes]

That is not a very informative error message :/
The spades executable is being called correctly in the container so we can exclude that. IDK if that error is related with memory. There's many people complaining about the say execution error code same error but from what I saw, no one explaining why it is happening.

[ramirma]

drpturner what is the OS you are running? We seem to have had problems in the past for reasons that remained unclarified while running some images built on different OSs than the one the user is running. This problem was restricted to singularity.