Add a script for comparing planemo list_repos output with a toolset

[mvdbeek]

planemo list_repos is a new command added in galaxyproject/planemo#982

[mvdbeek]

https://gist.github.com/mvdbeek/efb68e10ed8b632cf6be5f31903f0e70 is the output of this script for usegalaxy.org.
We're missing 510 IUC repos in the usegalaxy.org list. Not sure if we want to add all of them ?

[mvdbeek]

https://gist.github.com/mvdbeek/d4e1570dfbaddbc0537858bcd5977ecd is the list for https://github.com/usegalaxy-eu/usegalaxy-eu-tools, which is missing just 270 repos

[natefoo]

We ought to install at least the (set) difference between org and eu.

[natefoo]

Getting these in to sections is still going to be a manual process for now I guess. Should we consider aligning TS categories with usegalaxy.* sections?

[mvdbeek]

Want me to open a PR with that ? Will one PR work fine or should I split it up ?

[mvdbeek]

Should we consider aligning TS categories with usegalaxy.* sections?

That would be a neat way of at least getting the new repos in as they are published.

[natefoo]

I'd suggest breaking it up just to avoid having the entire super-long process fail due to some transient error.

[natefoo]

As an intermediate step to changing TS sections, could we add a usegalaxy.* section name/ID to the .shed.yml?

[mvdbeek]

Yeah, that should work. I'll start with the usegalaxy.eu diff ... though they might also have different sections ?

[mvdbeek]

That's what I've parsed out so far, but I gotta run.

Annotation:
- name: abricate
  owner: iuc
- name: augustus
  owner: bgruening
- name: augustus_training
  owner: bgruening
- name: bp_genbank2gff3
  owner: iuc
- name: blastxml_to_gapped_gff3
  owner: iuc
- name: busco
  owner: iuc
- name: exonerate
  owner: iuc
- name: fargene
  owner: iuc
- name: fgsea
  owner: iuc
- name: goslimmer
  owner: iuc
- name: goenrichment
  owner: iuc
- name: gprofiler_snpense
  owner: iuc
- name: gprofiler_gost
  owner: iuc
- name: gprofiler_random
  owner: iuc
- name: gprofiler_convert
  owner: iuc
- name: gprofiler_orth
  owner: iuc
- name: gubbins
  owner: iuc
- name: jcvi_gff_stats
  owner: iuc
- name: kobas
  owner: iuc
- name: maker_map_ids
  owner: iuc
- name: maker
  owner: iuc
- name: mlst
  owner: iuc
- name: progressivemauve
  owner: iuc
- name: repeat_masker
  owner: bgruening
- name: roary
  owner: iuc
- name: snap_training
  owner: iuc
- name: socru
  owner: iuc
- name: spaln
  owner: iuc
- name: tbprofiler
  owner: iuc
- name: tb_variant_filter
  owner: iuc
- name: tbvcfreport
  owner: iuc
- name: tetyper
  owner: iuc
- name: transtermhp
  owner: iuc
Assembly:
- name: abyss
  owner: iuc
- name: disco
  owner: iuc
- name: idba_hybrid
  owner: iuc
- name: idba_tran
  owner: iuc
- name: idba_ud
  owner: iuc
- name: megahit
  owner: iuc
- name: metaspades
  owner: nml
- name: miniasm
  owner: iuc
- name: pilon
  owner: iuc
- name: rnaspades
  owner: iuc
- name: shovill
  owner: iuc
- name: spades
  owner: nml
- name: velvetoptimiser
  owner: simon-gladman
Collection Operations:
- name: collection_element_identifiers
  owner: iuc
Convert Formats:
- name: bax2bam
  owner: iuc
- name: crossmap_bam
  owner: iuc
- name: crossmap_bed
  owner: iuc
- name: crossmap_wig
  owner: iuc
- name: crossmap_vcf
  owner: iuc
- name: crossmap_gff
  owner: iuc
- name: gfa_to_fa
  owner: iuc
- name: gtftobed12
  owner: iuc
- name: ucsc_twobittofa
  owner: iuc
DNA Metabarcoding:
- name: obi_clean
  owner: iuc
- name: obi_annotate
  owner: iuc
- name: obi_stat
  owner: iuc
- name: obi_tab
  owner: iuc
- name: obi_sort
  owner: iuc
- name: obi_ngsfilter
  owner: iuc
- name: obi_uniq
  owner: iuc
- name: obi_grep
  owner: iuc
- name: obi_convert
  owner: iuc
- name: obi_illumina_pairend
  owner: iuc
Epigenetics:
- name: moabs
  owner: iuc
Evolution:
- name: codeml
  owner: iuc
- name: iqtree
  owner: iuc
- name: structure
  owner: iuc
Expression Tools:
- name: compose_text_param
  owner: iuc
Extract Features:
- name: fraggenescan
  owner: iuc
FASTA/FASTQ:
- name: fasta_stats
  owner: iuc
- name: filtlong
  owner: iuc
- name: pear
  owner: iuc
- name: sickle
  owner: iuc
- name: ucsc_fasplit
  owner: iuc
- name: umi_tools_count
  owner: iuc
FASTQ Quality Control:
- name: prinseq
  owner: iuc
Gemini:
- name: gemini_set_somatic
  owner: iuc
- name: gemini_interactions
  owner: iuc
- name: gemini_stats
  owner: iuc
- name: gemini_gene_wise
  owner: iuc
- name: gemini_query
  owner: iuc
- name: gemini_windower
  owner: iuc
- name: gemini_db_info
  owner: iuc
- name: gemini_fusions
  owner: iuc
- name: gemini_load
  owner: iuc
- name: gemini_inheritance
  owner: iuc
- name: gemini_annotate
  owner: iuc
- name: gemini_lof_sieve
  owner: iuc
- name: gemini_qc
  owner: iuc
- name: gemini_burden
  owner: iuc
- name: gemini_roh
  owner: iuc
- name: gemini_actionable_mutations
  owner: iuc
- name: gemini_amend
  owner: iuc
- name: gemini_pathways
  owner: iuc
Genome Diversity:
- name: fasttree
  owner: iuc
Genome editing:
- name: crispr_studio
  owner: iuc
Get Data:
- name: ebi_metagenomics_run_downloader
  owner: iuc
- name: ebi_search_rest_results
  owner: iuc
- name: gdcwebapp
  owner: iuc
- name: ncbi_acc_download
  owner: iuc
Graph/Display Data:
- name: bandage
  owner: iuc
- name: charts
  owner: iuc
- name: export2graphlan
  owner: iuc
- name: ggplot2_violin
  owner: iuc
- name: ggplot2_point
  owner: iuc
- name: ggplot2_histogram
  owner: iuc
- name: ggplot2_heatmap
  owner: iuc
- name: ggplot2_pca
  owner: iuc
- name: intervene
  owner: iuc
- name: jbrowse
  owner: iuc
- name: newick_utils
  owner: iuc
- name: pathview
  owner: iuc
- name: pygenometracks
  owner: iuc
- name: tsne
  owner: iuc
- name: volcanoplot
  owner: iuc
HiCExplorer:
- name: hicexplorer_hicplotmatrix
  owner: bgruening
- name: hicexplorer_hicbuildmatrix
  owner: bgruening
- name: hicexplorer_hicaverageregions
  owner: bgruening
- name: hicexplorer_hicmergeloops
  owner: bgruening
- name: hicexplorer_hicplotsvl
  owner: bgruening
- name: hicexplorer_hicnormalize
  owner: bgruening
- name: hicexplorer_hicadjustmatrix
  owner: bgruening
- name: hicexplorer_chicaggregatestatistic
  owner: bgruening
- name: hicexplorer_hiccorrectmatrix
  owner: bgruening
- name: hicexplorer_hicplotviewpoint
  owner: bgruening
- name: hicexplorer_hicplotdistvscounts
  owner: bgruening
- name: hicexplorer_hiccorrelate
  owner: bgruening
- name: hicexplorer_hicplotaverageregions
  owner: bgruening
- name: hicexplorer_hicfindrestrictionsites
  owner: bgruening
- name: hicexplorer_hicmergematrixbins
  owner: bgruening
- name: hicexplorer_hicconvertformat
  owner: bgruening
- name: hicexplorer_hiccompartmentspolarization
  owner: bgruening
- name: hicexplorer_hicpca
  owner: bgruening
- name: hicexplorer_hicaggregatecontacts
  owner: bgruening
- name: hicexplorer_chicplotviewpoint
  owner: bgruening
- name: hicexplorer_chicqualitycontrol
  owner: bgruening
- name: hicexplorer_hicdetectloops
  owner: bgruening
- name: hicexplorer_hiccomparematrices
  owner: bgruening
- name: hicexplorer_chicdifferentialtest
  owner: bgruening
- name: hicexplorer_hictransform
  owner: bgruening
- name: hicexplorer_hicsummatrices
  owner: bgruening
- name: hicexplorer_hicfindtads
  owner: bgruening
- name: hicexplorer_chicviewpoint
  owner: bgruening
- name: hicexplorer_hicinfo
  owner: bgruening
- name: hicexplorer_chicviewpointbackgroundmodel
  owner: bgruening
- name: hicexplorer_chicsignificantinteractions
  owner: bgruening
- name: hicexplorer_hicquickqc
  owner: bgruening
- name: hicexplorer_hicvalidatelocations
  owner: bgruening
Mapping:
- name: bwameth
  owner: iuc
- name: minimap2
  owner: iuc
- name: rna_starsolo
  owner: iuc
Metagenomic Analysis:
- name: cat_add_names
  owner: iuc
- name: cat_prepare
  owner: iuc
- name: cat_contigs
  owner: iuc
- name: cat_summarise
  owner: iuc
- name: cat_bins
  owner: iuc
- name: dada2_plotqualityprofile
  owner: iuc
- name: dada2_plotcomplexity
  owner: iuc
- name: dada2_assigntaxonomyaddspecies
  owner: iuc
- name: dada2_removebimeradenovo
  owner: iuc
- name: dada2_filterandtrim
  owner: iuc
- name: dada2_dada
  owner: iuc
- name: dada2_learnerrors
  owner: iuc
- name: dada2_makesequencetable
  owner: iuc
- name: dada2_mergepairs
  owner: iuc
- name: dada2_seqcounts
  owner: iuc
- name: bio_hansel
  owner: iuc
- name: khmer_partition
  owner: iuc
- name: khmer_extract_partitions
  owner: iuc
- name: khmer_count_median
  owner: iuc
- name: khmer_filter_below_abundance_cutoff
  owner: iuc
- name: khmer_normalize_by_median
  owner: iuc
- name: khmer_abundance_distribution_single
  owner: iuc
- name: khmer_filter_abundance
  owner: iuc
- name: khmer_abundance_distribution
  owner: iuc
- name: maxbin2
  owner: mbernt
- name: nonpareil
  owner: iuc
- name: plasflow
  owner: iuc
- name: valet
  owner: iuc
Motif Tools:
- name: meme_psp_gen
  owner: iuc
- name: meme_chip
  owner: iuc
Multiple Alignments:
- name: msaboot
  owner: iuc
- name: mummer_delta_filter
  owner: iuc
- name: mummer_mummer
  owner: iuc
- name: mummer_dnadiff
  owner: iuc
- name: mummer_show_coords
  owner: iuc
- name: mummer_mummerplot
  owner: iuc
- name: mummer_nucmer
  owner: iuc
- name: pipelign
  owner: iuc
- name: sina
  owner: iuc
NCBI Blast:
- name: kraken2
  owner: iuc
Nanopore:
- name: nanoplot
  owner: iuc
- name: porechop
  owner: iuc
Peak Calling:
- name: genrich
  owner: iuc
- name: zerone
  owner: iuc
Phylogenetics:
- name: hyphy_bgm
  owner: iuc
- name: hyphy_fade
  owner: iuc
- name: hyphy_gard
  owner: iuc
- name: hyphy_sm19
  owner: iuc
- name: hyphy_absrel
  owner: iuc
- name: hyphy_busted
  owner: iuc
- name: hyphy_meme
  owner: iuc
- name: hyphy_slac
  owner: iuc
- name: hyphy_relax
  owner: iuc
- name: hyphy_fubar
  owner: iuc
- name: hyphy_fel
  owner: iuc
- name: phyml
  owner: iuc
- name: snp_dists
  owner: iuc
- name: snp_sites
  owner: iuc
Proteomics:
- name: megahit_contig2fastg
  owner: iuc
RAD-seq:
- name: stacks_ustacks
  owner: iuc
- name: stacks_sstacks
  owner: iuc
- name: stacks_genotypes
  owner: iuc
- name: stacks_stats
  owner: iuc
- name: stacks_populations
  owner: iuc
- name: stacks_refmap
  owner: iuc
- name: stacks_denovomap
  owner: iuc
- name: stacks_assembleperead
  owner: iuc
- name: stacks_clonefilter
  owner: iuc
- name: stacks_cstacks
  owner: iuc
- name: stacks_procrad
  owner: iuc
- name: stacks_rxstacks
  owner: iuc
- name: stacks_pstacks
  owner: iuc
- name: stacks2_shortreads
  owner: iuc
- name: stacks2_ustacks
  owner: iuc
- name: stacks2_sstacks
  owner: iuc
- name: stacks2_tsv2bam
  owner: iuc
- name: stacks2_gstacks
  owner: iuc
- name: stacks2_populations
  owner: iuc
- name: stacks2_refmap
  owner: iuc
- name: stacks2_kmerfilter
  owner: iuc
- name: stacks2_denovomap
  owner: iuc
- name: stacks2_clonefilter
  owner: iuc
- name: stacks2_cstacks
  owner: iuc
- name: stacks2_procrad
  owner: iuc
RNA Analysis:
- name: crosscontamination_barcode_filter
  owner: iuc
- name: deg_annotate
  owner: iuc
- name: egsea
  owner: iuc
- name: feelnc
  owner: iuc
- name: heinz
  owner: iuc
- name: idr
  owner: iuc
- name: masigpro
  owner: iuc
- name: pizzly
  owner: iuc
- name: qualimap_rnaseq
  owner: iuc
- name: qualimap_counts
  owner: iuc
- name: ruvseq
  owner: iuc
- name: scpipe
  owner: iuc
- name: seurat
  owner: iuc
- name: slamdunk
  owner: iuc
- name: transdecoder
  owner: iuc
- name: trinity_run_de_analysis
  owner: iuc
- name: trinity_super_transcripts
  owner: iuc
- name: trinity
  owner: iuc
- name: trinity_abundance_estimates_to_matrix
  owner: iuc
- name: trinity_contig_exn50_statistic
  owner: iuc
- name: trinity_align_and_estimate_abundance
  owner: iuc
- name: trinity_filter_low_expr_transcripts
  owner: iuc
- name: trinotate
  owner: iuc
- name: tximport
  owner: iuc
SAM/BAM:
- name: qualimap_multi_bamqc
  owner: iuc
- name: qualimap_bamqc
  owner: iuc
Single-cell:
- name: anndata_import
  owner: iuc
- name: anndata_export
  owner: iuc
- name: anndata_inspect
  owner: iuc
- name: anndata_manipulate
  owner: iuc
- name: modify_loom
  owner: iuc
- name: dropletutils
  owner: iuc
- name: raceid_filtnormconf
  owner: iuc
- name: raceid_trajectory
  owner: iuc
- name: raceid_clustering
  owner: iuc
- name: raceid_inspectclusters
  owner: iuc
- name: raceid_inspecttrajectory
  owner: iuc
- name: scanpy_remove_confounders
  owner: iuc
- name: scanpy_cluster_reduce_dimension
  owner: iuc
- name: scanpy_inspect
  owner: iuc
- name: scanpy_normalize
  owner: iuc
- name: scanpy_filter
  owner: iuc
- name: scanpy_plot
  owner: iuc
- name: scater_normalize
  owner: iuc
- name: scater_plot_dist_scatter
  owner: iuc
- name: scater_plot_exprs_freq
  owner: iuc
- name: scater_plot_pca
  owner: iuc
- name: scater_create_qcmetric_ready_sce
  owner: iuc
- name: scater_plot_tsne
  owner: iuc
- name: scater_filter
  owner: iuc
Statistics:
- name: graphembed
  owner: iuc
Text Manipulation:
- name: column_order_header_sort
  owner: iuc
- name: column_remove_by_header
  owner: iuc
- name: gff3_rebase
  owner: iuc
- name: jq
  owner: iuc
- name: reshape2_cast
  owner: iuc
- name: reshape2_melt
  owner: iuc
- name: table_compute
  owner: iuc
Variant Calling:
- name: basil
  owner: iuc
- name: bcftools_plugin_color_chrs
  owner: iuc
- name: bcftools_plugin_frameshifts
  owner: iuc
- name: exomedepth
  owner: crs4
- name: gatk4_mutect2
  owner: iuc
- name: mixcr_analyze
  owner: iuc
- name: ococo
  owner: iuc
- name: varscan_copynumber
  owner: iuc
- name: varscan_mpileup
  owner: iuc
- name: varscan_somatic
  owner: iuc
- name: snpsift_dbnsfp
  owner: iuc
- name: snpsift_genesets
  owner: iuc

[mvdbeek]

OK, I've created PRs for all the repos that have sections that already exist on usegalaxy.org and that are installed on usegalaxy.eu. Do we want to mirror the remaining sections that eu has and we don't, or do we want to do some more manual curation?

[natefoo]

Thanks!

Do we want to mirror the remaining sections that eu has and we don't

Most likely we do, at least for IUC tools. What sections are we missing?

[mvdbeek]

DNA Metabarcoding
Epigenetics
Evolution
Gemini
Genome editing
HiCExplorer
Peak Calling
Phylogenetics
Proteomics
RAD-seq
Single-cell

HiCExplorer could go to chromosome_conformation if we don't want to mirror the sections exactly.
Peak Calling overlaps with ChIP-seq but is not a subset (there are peak callers for other techniques in that section)

[natefoo]

Sounds good to me. We could also put the ChIP-seq tools in both sections?