Skip to content
This repository has been archived by the owner on Dec 14, 2023. It is now read-only.

Fix solr collections #842

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

Fix solr collections #842

wants to merge 18 commits into from

Commits on Mar 6, 2022

  1. Make no_dedup_sentences the default for extract-and-vector 

    	(can be overridden via MC_NO_DEDUP_SENTENCES env var)

apps/common/src/python/mediawords/util/config/__init__.py: add env_bool function
apps/extract-and-vector/bin/extract_and_vector_worker.py: honor MC_NO_DEDUP_SENTENCES
    @philbudne
    philbudne committed Mar 6, 2022
    Configuration menu
    Copy the full SHA
    815042a View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2022

  1. common/Dockerfile: skip jieba.cache creation; makes empty root owned … 

    …file
    @philbudne
    philbudne committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    c77dd18 View commit details
    Browse the repository at this point in the history

  2. solr-base/Dockerfile: try cloning mediacloud config as mediacloud64

    @philbudne
    philbudne committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    f37b7f9 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2022

  1. common/Dockerfile: reenable jieba.cache creation & chown it

    @philbudne
    philbudne committed Mar 23, 2022
    Configuration menu
    Copy the full SHA
    e1be9a4 View commit details
    Browse the repository at this point in the history

  2. apps/common/src/python/mediawords/solr/request.py: add/use SOLR_COLLE… 

    …CTION

Set to "mediacloud2" (solr alias!)
    @philbudne
    philbudne committed Mar 23, 2022
    Configuration menu
    Copy the full SHA
    1868296 View commit details
    Browse the repository at this point in the history

  3. apps/import-solr-data/src/perl/MediaWords/Solr/Dump.pm: speedups for … 

    …processing backlog

up FETCH_BLOCK_SIZE from 100 to 200:
ammortizes citus connection startup time by processing fetching larger blocks of text.

Honor skip_update_snapshot option (defaults to true):
skip setting snapshots.searchable=true
    @philbudne
    philbudne committed Mar 23, 2022
    Configuration menu
    Copy the full SHA
    4ab3729 View commit details
    Browse the repository at this point in the history

  4. add apps/solr-base/src/solr/aliases.json with "mediacloud2" solr alias

    @philbudne
    philbudne committed Mar 23, 2022
    Configuration menu
    Copy the full SHA
    791248a View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2022

  1. apps/common/src/requirements.txt: force MarkupSafe==2.0.1 

    (Jinja2 2.11.3 can't cope with the new MarkupSafe 2.1.1)
    @philbudne
    philbudne committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    c4fb3d6 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2022

  1. solr-zookeeper: preload aliases.json into zookeeper

    @philbudne
    philbudne committed Mar 27, 2022
    Configuration menu
    Copy the full SHA
    5845a61 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2022

  1. apps/postgresql-server/bin/apply_migrations.sh: increase PGCTL_START_… 

    …TIMEOUT to 3hrs
    @philbudne
    philbudne committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    40391ed View commit details
    Browse the repository at this point in the history

Commits on May 7, 2022

  1. postgresql-pgbouncer/conf/pgbounder.init: 

    PG server running on postgresql EC2 server w/o docker
    @philbudne
    philbudne committed May 7, 2022
    Configuration menu
    Copy the full SHA
    f395116 View commit details
    Browse the repository at this point in the history

  2. pgbouncer.ini: use postgresql server ip

    @philbudne
    philbudne committed May 7, 2022
    Configuration menu
    Copy the full SHA
    76be844 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2022

  1. apps/webapp-api/src/perl/MediaWords/Controller/Api/V2/Timespans.pm 

    Removed extra AND to try to fix mediacloud#833 as suggested by Rahul.
    @philbudne
    philbudne committed May 18, 2022
    Configuration menu
    Copy the full SHA
    bf74554 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2022

  1. fix solr query on multiple collections

    @thepsalmist
    thepsalmist committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    6654fbb View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2022

  1. refactor merge solr function

    @thepsalmist
    thepsalmist committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    e6bc74d View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2023

  1. fix solr response schema

    @thepsalmist
    thepsalmist committed Feb 9, 2023
    Configuration menu
    Copy the full SHA
    bc3c626 View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2023

  1. update solr merge function

    @thepsalmist
    thepsalmist committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    e8bb5a8 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2023

  1. fix solr merge buckets duplicat date values

    @thepsalmist
    thepsalmist committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    cb168c9 View commit details
    Browse the repository at this point in the history