Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resulting Image（png） Generation Issues #360

Open
RADIOMUMM opened this issue Mar 25, 2024 · 12 comments
Open

Resulting Image（png） Generation Issues #360

RADIOMUMM opened this issue Mar 25, 2024 · 12 comments

Comments

@RADIOMUMM
Copy link

Hello,

  • What parameter do I use to ensure that the following 4 images are generated in the results folder?

qcpic1="%s/result/01.QC/%s_cleandata/LengthvsQualityScatterPlot_kde.png"%(output,samplename)
qcpic2="%s/result/01.QC/%s_cleandata/Non_WeightedHistogramReadlength.png"%(output,samplename)
qcpic3="%s/result/01.QC/%s_cleandata/Non_WeightedLogTransformed_HistogramReadlength.png"%(output,samplename)
qcpic4="%s/result/01.QC/%s_cleandata/Yield_By_Length.png"%(output,samplename)

  • Now the command I use is:
    python3 NanoPlot.py -t 20 --N50 --dpi 300 --fastq test.fq.gz --title test -o %s/result/01.QC/%s

  • I've tried adding the --plots and --legacy parameters, but they don't generate the same results every time.
@wdecoster
Copy link
Owner

they don't generate the same results every time

Could you elaborate on which files you get and which you are missing?

@RADIOMUMM
Copy link
Author

RADIOMUMM commented Mar 26, 2024 •
edited
Loading

I used the command mentioned above and ran it twice in a row, using NanoFilt to intercept the Reads in between.
(NanoPlot --> NanoFilt -->NanoPlot)

First run of NanoPlot (rawdata):

  • 1.LengthvsQualityScatterPlot_dot.png
    -2.LengthvsQualityScatterPlot_kde.png
    -3.Non_weightedHistogramReadlength.png
    -4.Non_weightedLogTransformed_HistogramReadlength.png
    -5.WeightedHistogramReadlength.png
    -6.WeightedLogTransformed_HistogramReadlength.png
    -7.Yield_By_Length.png

Second run of NanoPlot (cleandata)：

  • 1.LengthvsQualityScatterPlot_dot.png
    -2.LengthvsQualityScatterPlot_kde.png
    -3.Non_weightedHistogramReadlength.png
    -4.Non_weightedLogTransformed_HistogramReadlength.png
    -6.Yield_By_Length.png

This doesn't happen every time; sometimes the results are the same for both runs.

@wdecoster
Copy link
Owner

I cannot think of any reason for that to happen.

@RADIOMUMM
Copy link
Author

No Yield_By_Length.png file was generated this time

(base)[root@418b066513c2 01.QC]# ll test_TB20009174_nocut_cleandata
total 3208
-rw-r--r-- 1 root root 487796 Mar 26 08:22 LengthvsQualityScatterPlot_dot.html
-rw-r--r-- 1 root root 44141 Mar 26 08:22 LengthvsQualityScatterPlot_dot.png
-rw-r--r-- 1 root root 725753 Mar 26 08:22 LengthvsQualityScatterPlot_kde.html
-rw-r--r-- 1 root root 63994 Mar 26 08:22 LengthvsQualityScatterPlot_kde.png
-rw-r--r-- 1 root root 1484310 Mar 26 08:22 NanoPlot-report.html
-rw-r--r-- 1 root root 3778 Mar 26 08:22 NanoPlot_20240326_0811.log
-rw-r--r-- 1 root root 841 Mar 26 08:15 NanoStats.txt
-rw-r--r-- 1 root root 844 Mar 26 08:15 NanoStats_post_filtering.txt
-rw-r--r-- 1 root root 13250 Mar 26 08:16 Non_weightedHistogramReadlength.html
-rw-r--r-- 1 root root 33020 Mar 26 08:16 Non_weightedHistogramReadlength.png
-rw-r--r-- 1 root root 15029 Mar 26 08:16 Non_weightedLogTransformed_HistogramReadlength.html
-rw-r--r-- 1 root root 47257 Mar 26 08:16 Non_weightedLogTransformed_HistogramReadlength.png
-rw-r--r-- 1 root root 14333 Mar 26 08:15 WeightedHistogramReadlength.html
-rw-r--r-- 1 root root 45382 Mar 26 08:15 WeightedHistogramReadlength.png
-rw-r--r-- 1 root root 17818 Mar 26 08:15 WeightedLogTransformed_HistogramReadlength.html
-rw-r--r-- 1 root root 43669 Mar 26 08:16 WeightedLogTransformed_HistogramReadlength.png
-rw-r--r-- 1 root root 198366 Mar 26 08:16 Yield_By_Length.html
-rw-r--r-- 1 root root 705 Mar 26 08:35 cyclone_qc.txt

(base)[root@418b066513c2 01.QC]# ll test_TB20009174_nocut_rawdata
total 3196
-rw-r--r-- 1 root root 483754 Mar 26 08:00 LengthvsQualityScatterPlot_dot.html
-rw-r--r-- 1 root root 48899 Mar 26 08:00 LengthvsQualityScatterPlot_dot.png
-rw-r--r-- 1 root root 719694 Mar 26 08:00 LengthvsQualityScatterPlot_kde.html
-rw-r--r-- 1 root root 84935 Mar 26 08:00 LengthvsQualityScatterPlot_kde.png
-rw-r--r-- 1 root root 1451376 Mar 26 08:00 NanoPlot-report.html
-rw-r--r-- 1 root root 3474 Mar 26 08:00 NanoPlot_20240326_0755.log
-rw-r--r-- 1 root root 845 Mar 26 08:00 NanoStats.txt
-rw-r--r-- 1 root root 13304 Mar 26 08:00 Non_weightedHistogramReadlength.html
-rw-r--r-- 1 root root 32855 Mar 26 08:00 Non_weightedHistogramReadlength.png
-rw-r--r-- 1 root root 15041 Mar 26 08:00 Non_weightedLogTransformed_HistogramReadlength.html
-rw-r--r-- 1 root root 43907 Mar 26 08:00 Non_weightedLogTransformed_HistogramReadlength.png
-rw-r--r-- 1 root root 14379 Mar 26 08:00 WeightedHistogramReadlength.html
-rw-r--r-- 1 root root 41574 Mar 26 08:00 WeightedHistogramReadlength.png
-rw-r--r-- 1 root root 17786 Mar 26 08:00 WeightedLogTransformed_HistogramReadlength.html
-rw-r--r-- 1 root root 39531 Mar 26 08:00 WeightedLogTransformed_HistogramReadlength.png
-rw-r--r-- 1 root root 177708 Mar 26 08:00 Yield_By_Length.html
-rw-r--r-- 1 root root 41155 Mar 26 08:00 Yield_By_Length.png
-rw-r--r-- 1 root root 707 Mar 26 08:35 cyclone_qc.txt

@RADIOMUMM
Copy link
Author

I cannot think of any reason for that to happen.
It's so weird.

微信截图_20240326172120

@wdecoster
Copy link
Owner

Can you share the log file of a run that failed to generate a png plot?

@RADIOMUMM
Copy link
Author

Can you share the log file of a run that failed to generate a png plot?

2024-03-26 08:11:31,873 NanoPlot 1.42.0 started with arguments Namespace(N50=True, alength=False, bam=None, barcoded=False, color='#4CB391', colormap='Greens', cram=None, downsample=None, dpi=300, drop_outliers=False, fasta=None, fastq=['/Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata_trimmed-reads.fastq.gz'], fastq_minimal=None, fastq_rich=None, feather=None, font_scale=1, format=['png'], hide_stats=False, huge=False, info_in_report=False, legacy=None, listcolormaps=False, listcolors=False, loglength=False, maxlength=None, minlength=None, minqual=None, no_N50=False, no_static=False, no_supplementary=False, only_report=False, outdir='/Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata', path='/Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/', percentqual=False, pickle=None, plots=['kde', 'dot'], prefix='', raw=False, readtype='1D', runtime_until=None, store=False, summary=None, threads=20, title='test_TB20009174_nocut_cleandata', tsv_stats=False, ubam=None, verbose=False)
2024-03-26 08:11:31,873 Python version is: 3.8.18 (default, Sep 11 2023, 13:40:15) [GCC 11.2.0]
2024-03-26 08:11:31,885 Nanoget: Starting to collect statistics from plain fastq file.
2024-03-26 08:11:31,885 Nanoget: Decompressing gzipped fastq /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata_trimmed-reads.fastq.gz
2024-03-26 08:15:57,025 Reduced DataFrame memory usage from 4.4492950439453125Mb to 4.4492950439453125Mb
2024-03-26 08:15:57,069 Nanoget: Gathered all metrics of 291581 reads
2024-03-26 08:15:57,213 Calculated statistics
2024-03-26 08:15:57,213 Using sequenced read lengths for plotting.
2024-03-26 08:15:57,222 Removed 29 artefactual reads with very short length and very high quality.
2024-03-26 08:15:57,355 Calculated statistics
2024-03-26 08:15:57,356 NanoPlot: Valid color #4CB391.
2024-03-26 08:15:57,356 NanoPlot: Valid colormap Greens.
2024-03-26 08:15:57,381 NanoPlot: Creating length plots for Read length.
2024-03-26 08:15:57,381 NanoPlot: Using 291552 reads with read length N50 of 30095bp and maximum of 117406bp.
2024-03-26 08:15:59,533 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/WeightedHistogramReadlength as png (or png for --legacy)
2024-03-26 08:16:01,478 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/WeightedLogTransformed_HistogramReadlength as png (or png for --legacy)
2024-03-26 08:16:03,489 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/Non_weightedHistogramReadlength as png (or png for --legacy)
2024-03-26 08:16:05,567 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/Non_weightedLogTransformed_HistogramReadlength as png (or png for --legacy)
2024-03-26 08:22:49,945 No static plots are saved due to some kaleido problem:
2024-03-26 08:22:49,945 Transform failed with error code 1: Failed to serialize document: Uncaught
2024-03-26 08:22:49,946 Created length plots
2024-03-26 08:22:49,952 NanoPlot: Creating Read lengths vs Average read quality plots using 291552 reads.
2024-03-26 08:22:52,388 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/LengthvsQualityScatterPlot_dot as png (or png for --legacy)
2024-03-26 08:22:55,236 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/LengthvsQualityScatterPlot_kde as png (or png for --legacy)
2024-03-26 08:22:55,236 Created LengthvsQual plot
2024-03-26 08:22:55,237 Writing html report.
2024-03-26 08:22:55,258 Finished!

@wdecoster
Copy link
Owner

This is it:

2024-03-26 08:22:49,945 No static plots are saved due to some kaleido problem:
2024-03-26 08:22:49,945 Transform failed with error code 1: Failed to serialize document: Uncaught

This doesn't help us, but for some reason it failed to create the PNG image for a particular plot. But you still have the plot in HTML format, right?

@RADIOMUMM
Copy link
Author

This is it:

2024-03-26 08:22:49,945 No static plots are saved due to some kaleido problem:
2024-03-26 08:22:49,945 Transform failed with error code 1: Failed to serialize document: Uncaught

This doesn't help us, but for some reason it failed to create the PNG image for a particular plot. But you still have the plot in HTML format, right?

right

@hoelzer
Copy link

hoelzer commented Jun 4, 2024

Hey, I have the same problem: no PNGs are produced and the errors are always:

2024-06-04 10:04:00,961 No static plots are saved due to some kaleido problem:
2024-06-04 10:04:00,961 Transform failed with error code 1: Failed to serialize document: Uncaught

I am running on a SLURM cluster and have NanoPlot installed in a Singularity container. NanoPlot version is 1.42.0

@hoelzer
Copy link

hoelzer commented Jun 4, 2024

This might be related to

https://stackoverflow.com/questions/71118601/saving-a-plotly-image-not-working-with-kaleido-even-though-it-is-installed

I am still wondering if this has something to do with executing NanoPlot from a container and/or a HPC environment...

@hoelzer
Copy link

hoelzer commented Jun 4, 2024

Further research revelad that this might be also a problem with internet connection that Kaleido apparently needs to convert HTML to PNG. THis would explain my issue bc I work on a quite restrictive HPC with a lot of ports closed.

See:
plotly/Kaleido#108

Maybe it is possible to also configure Kaleido in NanoPlot to work offline?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wdecoster @hoelzer @RADIOMUMM