Update 0.4.5 (#180)

* rework in progress. * fix: correct limits for orionoid * fix: switch to comprehensions * fix: disable plex logging for id mismatches * time for sleep. rework still wip. * feat: parser works. needs more work. language needs a rewrite. disabled for now. * fix: overseerr bug on using external ids * fix: remove plex debug line for users * disable tvdb checks from listrr, overseerr, plex. needs reworked. * set torrentio to disabled by default. removed parse logs. raised torrentio limit slightly. * Set all default settings to disabled by default for onboarding * add extra logging attr. for debugging large groups of data. * feat: started status page rewrite * add dev branch builds with :dev tag (#165) * fix: listrr validation * rework in progress. * fix: correct limits for orionoid * fix: switch to comprehensions * fix: disable plex logging for id mismatches * time for sleep. rework still wip. * feat: parser works. needs more work. language needs a rewrite. disabled for now. * feat: frontend improvements (#158) * feat: added global debug for settings * feat: added dev to formDebug so it's always true in development but false in production * feat: added DEBUG & LOG to general settings * deps: switched svelte-sonner to shadcn customized toast component * refactor: renamed PlexDebridItem to IcebergItem and added changes made to /items * fix: fixed the wrong relative date in status page * docs: readme improvements * docs: readme improvements * docs: readme improvements * chore(deps): bump lucide-svelte from 0.303.0 to 0.307.0 in /frontend (#124) * refactor: componentized forms, soon will do same for fields too * Parse rewrite (#128) * Move parser to its own module * Add ORIGIN to env vars * Fix overseerr, watchlist, jackett validation. * Added more refined logic to parser module. * Set stage for testing * Add methods for individual checks * Update sort logic * Update default settings * Fix jackett. Begin to add title support for jackett. --------- Co-authored-by: Spoked <Spoked@localhost> Co-authored-by: Dreu LaVelle <dreu.lavelle@localhost> * feat: onboarding on the way ;), major refactoring of form related code * Simplified downloading logic and modified state matchine * fix typo in state machine and handle movie pathing correctly * Remove useless method * Temporary fix to test * Remove uncached stream hashes from item to avoid loop, some blacklisting logic could also be good * chore(deps-dev): bump @typescript-eslint/eslint-plugin in /frontend (#134) * chore(deps): bump lucide-svelte from 0.307.0 to 0.309.0 in /frontend (#133) * chore(deps-dev): bump @sveltejs/kit from 2.0.1 to 2.3.2 in /frontend (#132) * chore(deps): bump bits-ui from 0.13.0 to 0.14.0 in /frontend (#130) * chore(deps-dev): bump @sveltejs/adapter-node in /frontend (#138) * feat: some more onboarding and form improvements * Dev startup to disabling pickling * feat: Listrr Support Added (#136) * Start Listrr Feature * feat: Listrr ready for review. * small tweaks. rewrite coming later. --------- Co-authored-by: Spoked <Spoked@localhost> * Jackett rewrite (#139) * Add TorBox scraper * Add is_anime attribute to item * Rework Jackett to Keyword Queries. Added categories. Removed Torbox * Remove audio from parsing, it removed alot of good hits * fix movie scraping and modify response parsing logic to be more readable * fix: remove torbox module * remove audio from being parsed * remove more audio from parser * fix typo * fix: tidy audio and networks * small tweaks --------- Co-authored-by: Spoked <Spoked@localhost> Co-authored-by: Gaisberg <None> * Avoid [None] if empty content service * fix: handle bad quality manually in parser (#145) Co-authored-by: Spoked <Spoked@localhost> * deps: updated deps due to security updateS * feat: added more onboarding steps, some bugs also introduced * chore(deps-dev): bump @sveltejs/kit from 2.3.2 to 2.4.2 in /frontend (#156) * chore(deps-dev): bump prettier from 3.1.1 to 3.2.4 in /frontend (#155) * chore(deps): bump lucide-svelte from 0.309.0 to 0.314.0 in /frontend (#154) * chore(deps): bump bits-ui from 0.14.0 to 0.15.1 in /frontend (#153) * feat: minor changes * feat: deps change * feat: deps change * feat: onboarding MVP done --------- Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Spoked <[email protected]> Co-authored-by: Spoked <Spoked@localhost> Co-authored-by: Dreu LaVelle <dreu.lavelle@localhost> Co-authored-by: Gaisberg <None> * feat: frontend improvements (#159) * feat: added global debug for settings * feat: added dev to formDebug so it's always true in development but false in production * feat: added DEBUG & LOG to general settings * deps: switched svelte-sonner to shadcn customized toast component * refactor: renamed PlexDebridItem to IcebergItem and added changes made to /items * fix: fixed the wrong relative date in status page * docs: readme improvements * docs: readme improvements * docs: readme improvements * chore(deps): bump lucide-svelte from 0.303.0 to 0.307.0 in /frontend (#124) * refactor: componentized forms, soon will do same for fields too * Parse rewrite (#128) * Move parser to its own module * Add ORIGIN to env vars * Fix overseerr, watchlist, jackett validation. * Added more refined logic to parser module. * Set stage for testing * Add methods for individual checks * Update sort logic * Update default settings * Fix jackett. Begin to add title support for jackett. --------- Co-authored-by: Spoked <Spoked@localhost> Co-authored-by: Dreu LaVelle <dreu.lavelle@localhost> * feat: onboarding on the way ;), major refactoring of form related code * Simplified downloading logic and modified state matchine * fix typo in state machine and handle movie pathing correctly * Remove useless method * Temporary fix to test * Remove uncached stream hashes from item to avoid loop, some blacklisting logic could also be good * chore(deps-dev): bump @typescript-eslint/eslint-plugin in /frontend (#134) * chore(deps): bump lucide-svelte from 0.307.0 to 0.309.0 in /frontend (#133) * chore(deps-dev): bump @sveltejs/kit from 2.0.1 to 2.3.2 in /frontend (#132) * chore(deps): bump bits-ui from 0.13.0 to 0.14.0 in /frontend (#130) * chore(deps-dev): bump @sveltejs/adapter-node in /frontend (#138) * feat: some more onboarding and form improvements * Dev startup to disabling pickling * feat: Listrr Support Added (#136) * Start Listrr Feature * feat: Listrr ready for review. * small tweaks. rewrite coming later. --------- Co-authored-by: Spoked <Spoked@localhost> * Jackett rewrite (#139) * Add TorBox scraper * Add is_anime attribute to item * Rework Jackett to Keyword Queries. Added categories. Removed Torbox * Remove audio from parsing, it removed alot of good hits * fix movie scraping and modify response parsing logic to be more readable * fix: remove torbox module * remove audio from being parsed * remove more audio from parser * fix typo * fix: tidy audio and networks * small tweaks --------- Co-authored-by: Spoked <Spoked@localhost> Co-authored-by: Gaisberg <None> * Avoid [None] if empty content service * fix: handle bad quality manually in parser (#145) Co-authored-by: Spoked <Spoked@localhost> * deps: updated deps due to security updateS * feat: added more onboarding steps, some bugs also introduced * chore(deps-dev): bump @sveltejs/kit from 2.3.2 to 2.4.2 in /frontend (#156) * chore(deps-dev): bump prettier from 3.1.1 to 3.2.4 in /frontend (#155) * chore(deps): bump lucide-svelte from 0.309.0 to 0.314.0 in /frontend (#154) * chore(deps): bump bits-ui from 0.14.0 to 0.15.1 in /frontend (#153) * feat: minor changes * feat: deps change * feat: deps change * feat: onboarding MVP done * refactor: moved schemes into forms/helpers.ts and command menu improvements * refactor: switched to new font, changes made to all except status page * refactor: minor change, didn't get commit * fix: minor fix * feat: fixed git merge conflicts issue * feat: fixed status page font too --------- Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Spoked <[email protected]> Co-authored-by: Spoked <Spoked@localhost> Co-authored-by: Dreu LaVelle <dreu.lavelle@localhost> Co-authored-by: Gaisberg <None> * docs: minor improvements (#160) * docs: readme improvements * docs: readme improvements * docs: minor improvements (#161) * docs: readme improvements * docs: readme improvements * docs: readme improvements * docs: minor improvements (#162) * fix: correct parsing of external id's (#163) Co-authored-by: Dreu LaVelle <[email protected]> * fix: overseerr bug on using external ids * fix: remove plex debug line for users * disable tvdb checks from listrr, overseerr, plex. needs reworked. * set torrentio to disabled by default. removed parse logs. raised torrentio limit slightly. * Set all default settings to disabled by default for onboarding * add extra logging attr. for debugging large groups of data. * fix: listrr validation * feat: rewrite of status page almost done * add verbose logging in plex to debug looping * revert debug logging for plex. figured out looping issue. * add back plex log. add boilerplate for trakt content service. wip * add back plex log. add boilerplate for trakt content service. wip * feat: status page improvements (#169) * Feat/better status page (#170) * feat: status page improvements * feat: status page improvements * feat: status page improvements * feat: new settings (#176) * added more validation and logging * changed name of test_items module * refactor: edited minor things in settings (#177) * remove tzdata * refactor: edited minor things in settings (#177) * remove tzdata * fix typo on parser * increase ratelimits on second_limiters * Fix/parser/add attribute (#179) * increase ratelimits on second_limiters * Iceberg works. All scrapers working together. Symlinking works. --------- Co-authored-by: Spoked <Spoked@localhost> * add extra attrs to extended api endpoint * feat: status page improvements (#182) --------- Co-authored-by: Spoked <Spoked@localhost> Co-authored-by: Ayush Sehrawat <[email protected]> Co-authored-by: KingPin <[email protected]> Co-authored-by: Dreu LaVelle <[email protected]> Co-authored-by: Ayush Sehrawat <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dreu LaVelle <dreu.lavelle@localhost>
rivenmedia · Jan 29, 2024 · b3b816b · b3b816b
1 parent b155e60
commit b3b816b
Show file tree

Hide file tree

Showing 79 changed files with 1,502 additions and 687 deletions.
diff --git a/.github/workflows/docker-build-dev.yml b/.github/workflows/docker-build-dev.yml
@@ -0,0 +1,57 @@
+name: Docker dev branch Build and Push
+
+on:
+  push:
+    branches:
+      - dev
+  workflow_dispatch:
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      packages: write
+      security-events: write
+
+    steps:
+    - name: Docker Setup QEMU
+      uses: docker/setup-qemu-action@v3
+      id: qemu
+      with:
+        platforms: amd64,arm64
+
+    - name: Log into ghcr.io registry
+      uses: docker/login-action@v3
+      with:
+        registry: ghcr.io
+        username: ${{ github.repository_owner }}
+        password: ${{ secrets.GITHUB_TOKEN }}
+
+    - name: set lower case owner name
+      run: |
+        echo "GITHUB_OWNER_LC=${OWNER,,}" >>${GITHUB_ENV}
+      env:
+        OWNER: '${{ github.repository_owner }}'
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v3
+
+    - name: Log in to Docker Hub
+      uses: docker/login-action@v3
+      with:
+        username: ${{ secrets.DOCKER_HUB_USERNAME }}
+        password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}
+
+    - name: Build and push Docker image
+      uses: docker/build-push-action@v5
+      with:
+        # context: .
+        file: ./Dockerfile
+        platforms: linux/amd64,linux/arm64
+        push: true
+        cache-from: type=gha
+        cache-to: type=gha,mode=max
+        tags:  |
+            ghcr.io/${{ env.GITHUB_OWNER_LC }}/${{ github.event.repository.name }}:dev
+            docker.io/spoked/iceberg:dev
diff --git a/Dockerfile b/Dockerfile
@@ -14,12 +14,14 @@ LABEL name="Iceberg" \
       description="Iceberg Debrid Downloader" \
       url="https://github.com/dreulavelle/iceberg"
 
-RUN apk --update add python3 py3-pip bash shadow vim nano rclone && \
+RUN apk --update add --no-cache python3 py3-pip bash shadow && \
     rm -rf /var/cache/apk/*
 
 WORKDIR /iceberg
-ENV ORIGIN http://localhost:3000
-ARG ORIGIN=http://localhost:3000
+
+ARG ORIGIN
+ENV ORIGIN=${ORIGIN:-http://localhost:3000}
+
 EXPOSE 3000 8080
 
 # Frontend

diff --git a/README.md b/README.md
@@ -62,7 +62,7 @@ version: "3.8"
 services:
     iceberg:
         image: spoked/iceberg:latest
-        container_name: Iceberg
+        container_name: iceberg
         restart: unless-stopped
         environment:
             PUID: "1000"
@@ -72,6 +72,7 @@ services:
             - "3000:3000"
         volumes:
             - ./data:/iceberg/data
+            - /mnt:/mnt
 ```
 
 Then run `docker compose up -d` to start the container in the background. You can then access the web interface at `http://localhost:3000` or whatever port and origin you set in the `docker-compose.yml` file.

diff --git a/backend/controllers/items.py b/backend/controllers/items.py
@@ -31,7 +31,7 @@ async def get_extended_item_info(request: Request, item_id: str):
         raise HTTPException(status_code=404, detail="Item not found")
     return {
         "success": True,
-        "item": item.to_extended_dict(),  # Assuming this method exists
+        "item": item.to_extended_dict(),
     }
 
 
@@ -42,3 +42,14 @@ async def remove_item(request: Request, item: str):
         "success": True,
         "message": f"Removed {item}",
     }
+
+
+@router.get("/imdb/{imdb_id}")
+async def get_imdb_info(request: Request, imdb_id: str):
+    item = request.app.program.media_items.get_item_by_imdb_id(imdb_id)
+    if item is None:
+        raise HTTPException(status_code=404, detail="Item not found")
+    return {
+        "success": True,
+        "item": item.to_extended_dict()
+    }
diff --git a/backend/program/__init__.py b/backend/program/__init__.py
@@ -58,9 +58,13 @@ def validate(self):
         return all(service.initialized for service in self.core_manager.services)
 
     def stop(self):
-        for service in self.core_manager.services:
-            if getattr(service, "running", False):
-                service.stop()
-        self.pickly.stop()
-        settings.save()
-        self.running = False
+        try:
+            for service in self.core_manager.services:
+                if getattr(service, "running", False):
+                    service.stop()
+            self.pickly.stop()
+            settings.save()
+            self.running = False
+        except Exception as e:
+            logger.error("Iceberg stopping with exception: %s", e)
+            pass
diff --git a/backend/program/content/listrr.py b/backend/program/content/listrr.py
@@ -35,17 +35,32 @@ def __init__(self, media_items: MediaItemContainer):
         logger.info("Listrr initialized!")
 
     def validate_settings(self) -> bool:
+        """Validate Listrr settings."""
         if not self.settings.enabled:
             logger.debug("Listrr is set to disabled.")
             return False
         if self.settings.api_key == "" or len(self.settings.api_key) != 64:
-            logger.error("Listrr api key is not set.")
+            logger.error("Listrr api key is not set or invalid.")
+            return False
+        valid_list_found = False
+        for list_name, content_list in [('movie_lists', self.settings.movie_lists), 
+                                        ('show_lists', self.settings.show_lists)]:
+            if content_list is None or not any(content_list):
+                continue
+            for item in content_list:
+                if item == "" or len(item) != 24:
+                    return False
+            valid_list_found = True
+        if not valid_list_found:
+            logger.error("Both Movie and Show lists are empty or not set.")
             return False
         try:
             response = ping("https://listrr.pro/", additional_headers=self.headers)
+            if not response.ok:
+                logger.error(f"Listrr ping failed - Status Code: {response.status_code}, Reason: {response.reason}")
             return response.ok
-        except Exception:
-            logger.error("Listrr url is not reachable.")
+        except Exception as e:
+            logger.error(f"Listrr ping exception: {e}")
             return False
 
     def run(self):
@@ -56,7 +71,8 @@ def run(self):
         movie_items = self._get_items_from_Listrr("Movies", self.settings.movie_lists)
         show_items = self._get_items_from_Listrr("Shows", self.settings.show_lists)
         items = list(set(movie_items + show_items))
-        container = self.updater.create_items(items)
+        new_items = [item for item in items if item not in self.media_items]
+        container = self.updater.create_items(new_items)
         for item in container:
             item.set("requested_by", "Listrr")
         added_items = self.media_items.extend(container)
@@ -87,11 +103,11 @@ def _get_items_from_Listrr(self, content_type, content_lists):
                             imdb_id = item.imDbId
                             if imdb_id:
                                 unique_ids.add(imdb_id)
-                            elif content_type == "Shows" and item.tvDbId:
-                                imdb_id = get_imdbid_from_tvdb(item.tvDbId)
-                                if imdb_id:
-                                    unique_ids.add(imdb_id)
-                            elif content_type == "Movies" and item.tmDbId:
+                            # elif content_type == "Shows" and item.tvDbId:
+                            #     imdb_id = get_imdbid_from_tvdb(item.tvDbId)
+                            #     if imdb_id:
+                            #         unique_ids.add(imdb_id)
+                            if not imdb_id and content_type == "Movies" and item.tmDbId:
                                 imdb_id = get_imdbid_from_tmdb(item.tmDbId)
                                 if imdb_id:
                                     unique_ids.add(imdb_id)

diff --git a/backend/program/content/overseerr.py b/backend/program/content/overseerr.py
@@ -42,6 +42,11 @@ def validate_settings(self) -> bool:
                 additional_headers=self.headers,
                 timeout=15,
             )
+            if response.status_code >= 201:
+                logger.error(
+                    f"Overseerr ping failed - Status Code: {response.status_code}, Reason: {response.reason}"
+                )
+                return False
             return response.ok
         except Exception:
             logger.error("Overseerr url is not reachable.")
@@ -78,7 +83,6 @@ def _get_items_from_overseerr(self, amount: int):
                         ids.append(imdb_id)
                 else:
                     ids.append(item.media.imdbId)
-
         return ids
 
     def get_imdb_id(self, overseerr_item):
@@ -97,31 +101,26 @@ def get_imdb_id(self, overseerr_item):
             self.settings.url + f"/api/v1/{overseerr_item.mediaType}/{external_id}?language=en",
             additional_headers=self.headers,
         )
-        if response.is_ok and response.data.externalIds:
-            imdb_id = response.data.externalIds.imdbId
-            if imdb_id:
-                return imdb_id
-            elif not imdb_id and response.data.externalIds.tvdbId:
-                imdb_id = get_imdbid_from_tvdb(response.data.externalIds.tvdbId)
-                if imdb_id:
-                    logger.debug(
-                        "Could not find imdbId for %s but found it from tvdbId %s", 
-                        overseerr_item.title, 
-                        response.data.externalIds.tvdbId
-                        )
-                    return imdb_id
-            elif not imdb_id and response.data.externalIds.tmdbId:
-                imdb_id = get_imdbid_from_tmdb(response.data.externalIds.tmdbId)
-                if imdb_id:
-                    logger.debug(
-                        "Could not find imdbId for %s but found it from tmdbId %s", 
-                        overseerr_item.title, 
-                        response.data.externalIds.tmdbId
-                        )
-                    return imdb_id
-            self.not_found_ids.append(f"{id_extension}{external_id}")
-        title = getattr(response.data, "title", None) or getattr(
-            response.data, "originalName", None
-        )
-        logger.debug("Could not get imdbId for %s, or match with external id", title)
+        if not response.is_ok or not hasattr(response.data, "externalIds"):
+            logger.debug(f"Failed to fetch or no externalIds for {id_extension}{external_id}")
+            return None
+
+        title = getattr(response.data, "title", None) or getattr(response.data, "originalName", None)
+        imdb_id = getattr(response.data.externalIds, 'imdbId', None)
+        if imdb_id:
+            return imdb_id
+
+        # Try alternate IDs if IMDb ID is not available
+        # alternate_ids = [('tvdbId', get_imdbid_from_tvdb), ('tmdbId', get_imdbid_from_tmdb)]
+        alternate_ids = [('tmdbId', get_imdbid_from_tmdb)]
+        for id_attr, fetcher in alternate_ids:
+            external_id_value = getattr(response.data.externalIds, id_attr, None)
+            if external_id_value:
+                new_imdb_id = fetcher(external_id_value)
+                if new_imdb_id:
+                    logger.debug(f"Found imdbId for {title} from {id_attr}: {external_id_value}")
+                    return new_imdb_id
+
+        self.not_found_ids.append(f"{id_extension}{external_id}")
+        logger.debug(f"Could not get imdbId for {title}, or match with external id")
         return None
diff --git a/backend/program/content/plex_watchlist.py b/backend/program/content/plex_watchlist.py
@@ -1,7 +1,7 @@
 """Plex Watchlist Module"""
 from typing import Optional
 from pydantic import BaseModel
-from requests import ConnectTimeout
+from requests import ConnectTimeout, HTTPError
 from utils.request import get, ping
 from utils.logger import logger
 from utils.settings import settings_manager
@@ -29,29 +29,50 @@ def __init__(self, media_items: MediaItemContainer):
         self.media_items = media_items
         self.prev_count = 0
         self.updater = Trakt()
+        self.not_found_ids = []
 
     def validate_settings(self):
         if not self.settings.enabled:
             logger.debug("Plex Watchlists is set to disabled.")
             return False
         if self.settings.rss:
+            logger.info("Found Plex RSS URL. Validating...")
             try:
-                response = ping(self.settings.rss, timeout=15)
+                response = ping(self.settings.rss)
                 if response.ok:
                     self.rss_enabled = True
+                    logger.info("Plex RSS URL is valid.")
                     return True
                 else:
-                    logger.warn(f"Plex RSS URL is not reachable. Falling back to normal Watchlist.")
+                    logger.info(f"Plex RSS URL is not valid. Falling back to watching user Watchlist.")
                     return True
-            except Exception:
-                return False
+            except HTTPError as e:
+                if e.response.status_code in [404]:
+                    logger.warn("Plex RSS URL is Not Found. Falling back to watching user Watchlist.")
+                    return True
+                if e.response.status_code >= 400 and e.response.status_code <= 499:
+                    logger.warn(f"Plex RSS URL is not reachable. Falling back to watching user Watchlist.")
+                    return True
+                if e.response.status_code >= 500:
+                    logger.error(f"Plex is having issues validating RSS feed. Falling back to watching user Watchlist.")
+                    return True
+            except Exception as e:
+                logger.exception("Failed to validate Plex RSS URL: %s", e)
+                return True
         return True
 
     def run(self):
         """Fetch media from Plex Watchlist and add them to media_items attribute
         if they are not already there"""
         items = self._create_unique_list()
         new_items = [item for item in items if item not in self.media_items] or []
+        if len(new_items) == 0:
+            logger.debug("No new items found in Plex Watchlist")
+            return
+        for check in new_items:
+            if check is None:
+                new_items.remove(check)
+                self.not_found_ids.append(check)
         container = self.updater.create_items(new_items)
         for item in container:
             item.set("requested_by", "Plex Watchlist")
@@ -68,6 +89,9 @@ def run(self):
                 logger.info("Added %s", item.log_string)
         elif length > 5:
             logger.info("Added %s items", length)
+        if len(self.not_found_ids) >= 1 and len(self.not_found_ids) <= 5:
+            for item in self.not_found_ids:
+                logger.info("Failed to add %s", item)
 
     def _create_unique_list(self):
         """Create a unique list of items from Plex RSS and Watchlist"""
@@ -80,7 +104,7 @@ def _create_unique_list(self):
     def _get_items_from_rss(self) -> list:
         """Fetch media from Plex RSS Feed"""
         try:
-            response_obj = get(self.settings.rss, timeout=30)
+            response_obj = get(self.settings.rss, timeout=60)
             data = json.loads(response_obj.response.content)
             items = data.get("items", [])
             ids = [