TheTVDB v4 licensing and HAMA future #457
Comments
|
Thinking on this, but bringing @EndOfLine369 and @reconman into this. |
|
First, the lift of moving to TheMovieDB is probably extremely heavy on the Anime-Lists side of this. I could be wrong, but that is my first take. Second, it seems like they have decent metadata so if we do find a way to either A) incorporate TheMovieDB for TV shows in general or B) switch over -- I think we'll be in decent shape. Before committing to that, figuring out the mapping is critical. I do see in their metadata that they have TheTVDB ID as a field that can be filled out. We might be able to work with that? My biggest issue with other options Sonaar, Shoko, etc. is the Absolute Numbering and automatic season mapping that HAMA does so well. There are a few thoughts I have on this part to make sure we don't lose it.
Edit: Question - Do we ALL need a TVDB license or can this operate off of one? |
|
Keeping anime-lists compatible with TVDB AND TMDB is impossible in my opinion. If we want to accomplish that, we would need a mapping structure similar to thexem, which means
Maybe the folks at thexem are also thinking about the TVDB problem. I'm thinking that maybe thexem could include additional databases like IMDB and TMDB in their mappings, but the codebase is probably hard to maintain, so I don't expect them to be able to adjust easily. Additionally, it would restrict the number of users who can edit the mappings, since user registrations are closed. @thezoggy Since you're the only one maintaining the xem code base, what's your input on enhancing thexem with additional databases and on the TVDB deprecation? |
|
TheTVDB series121,161, movie153,237 person510,784 [https://thetvdb.com/search?] Comparison page TMDB/TVDB: Choices are
Get tvdbid 'Find' by tvdb id: Extras: |
Perhaps we can use this TheTVDB ID to bridge the gap between the three? On the one hand, $12/year doesn't truly bother me (aside from I disagree with what they're doing), especially if they make contributors free, then likely $0 option can be palatable. |
|
Issue is order of eps can be different and dual episodes can be split or not. We could just use my code and do a separate TMDB only agent, managing movies and series. That would allow to test TMDB, be useful for documentaries as like anime they mix movies and series in a single library and would minimise the time to port it to hama, and allow people to add what is not on TheMovieDb... |
|
I would rather hama somehow completely ditch thetvdb/themoviedb/imdb etc in my opinion. They are simply not made for anime. An anime agent should strive to use databases/apis specifically designed for anime, that means: MAL, AniDB, AniList etc etc. Specifically, the way TheTVDB defines anime in its databases is simply not the way the anime community does and it causes WAY too many problems. Anyway, that's my two cents. |
|
TheTVDB have episode summaries in multiple language when AniDB had no summary then now english only episode summaries... Not everybody is fan of AniDb numbering and there is more that one way in the 'Anime community' Furthermore we need an infrastructure that is the best we can do given the circumstances, as metadata can be added... Why are you not using the following agents (especially AniDb):
If you tell me the metadata is lacking on a single database, I would recommend to add the missing meta, anyway, that's my two cents. |
|
https://support.thetvdb.com/kb/faq.php?id=82
http://thexem.de/ is anime specific and could be used if TMDB matches the scene...
|
|
So......there are multiple items to discuss:
<name>Gintama'</name>
<mapping-list>
<mapping anidbseason="0" tvdbseason="0">;1-6;</mapping>
<mapping anidbseason="0" tvdbseason="0" start="1" end="6" offset="16"/>
</mapping-list>
<supplemental-info>
<studio>Sunrise</studio>
</supplemental-info>An additional "mapping-list-tsdb"/"mapping-tsdb" tags & "tsdbseason" attribute with info could be added. Apps should be looking for specific xml child tags by name so by adding a new named child I would think not interfere with any application. <mapping-list-tsdb>
<mapping-tsdb anidbseason="0" tsdbseason="0">;1-6;</mapping>
<mapping-tsdb anidbseason="0" tsdbseason="0" start="1" end="6" offset="16"/>
</mapping-list-tsdb>Or even just adding the new "mapping-tsdb" child tags into the existing "mapping-list" group w/diff "tsdbseason" attribute. This would be a little riskier as could potentially cause issues in apps plainly iterating over the "mapping-list" childs. Adding a different "mapping-tsdb" tag & attribute:
<mapping-list>
<mapping anidbseason="0" tvdbseason="0">;1-6;</mapping>
<mapping-tsdb anidbseason="0" tsdbseason="0">;1-6;</mapping>
</mapping-list>
OR having the same tag but different attribute
<mapping-list>
<mapping anidbseason="0" tvdbseason="0">;1-6;</mapping>
<mapping anidbseason="0" tsdbseason="0">;1-6;</mapping>
</mapping-list>
|
|
Since TheTVDV is mapped to anidb it is used in all modes (anidb-tvdb) but required for anidb2-5 and then provide summaries in other languages which and can't... Therefore for summaries in other languages and mapping it is pretty needed, especially for users that used the mapping modes There is a episode numbering field for TheTVDB or TMDb in new plex series agent, so there might be a way to map... TMDb have very precise rules for anime which should limit bickering and TVDB locking on an arbitrary order TMDb api does support 7 types for ordering
These mapping are interesting, and letting all these numberings to be chosen by the user worthwhile, but might warrant a new agent doing movies and series using TheTVDB (for documentaries, anime) or a subgroup [tsdb1-xxx] to [tsdb7-xxx] to allow for weird series to switch to TMDB series useful numbering... dunno about TVDB API v2 proxy or how to use API key, i do have one though |
I'm 100% with you here. I think we need to support this. I'll also try to contribute and get a free key.
@EndOfLine369 I was talking with someone yesterday bout the idea of quering TheMovieDB's API via the External ID field for TheTVDB. https://developers.themoviedb.org/3/getting-started/external-ids In my mind, we could use that to snapshot Anime-Lists with a 'tsdb' field. It might not be perfect, but will do a lot of the lifting. Alternatively/additionally, ASS/HAMA might be able to do the same, if Anime-Lists continues to stay updated. |
|
@ZeroQI, Good to know the different ordering options from TMDB are available. TVDB also provides different ordering output via "/series/{id}/episodes/{season-type}" now in their latest api. That's what you now see on the website post their last major upgrade. We just do it ourselves by sorting and pulling out the details to override ep number. For TMDB, I think we should keep it at only Official, Absolute, Arc, & Air Date like we currently do for TVDB. Arc would depend if their Arc groupings are reasonable or not and we have to just create our own mapping files like we did for TVDB4 for seasons & images. Just create the mirrors (as mention is item 6 above) of what we already provide from TVDB. New orders would make this work more complicated than this transition work should require. Other modes can be created later, if even wanted, after the addition of TMDB for other episode order options. Trying to separate out to a different agent doesn't seen realistic. It would require code to be duplicated from HAMA and then kept in sync. As well as would remove the ability to have multiple sources as options for fields. If kept all in one, it would even allow for TVDB & TMDB to be both possible source options for a series. Like episode titles from TMDB but episode summaries from TVDB. That's the whole major advantage of HAMA. Multiple sources in one agent 😁. If we can agree on the above, its just a matter then of the where/how we get the AniDB -> TMDB mapping. |
We could potentially setup a dump file like we do for missing episodes/summaries, and have it contain info on when there is a TVDB id but no TSDB id. If not this, what could you potentially be looking for from HAMA? Getting one off bulk update of TMDB ids from TVDB id for tsdb ids would get probably 50% firmly sorted as most of the entries as have no special TVDB mapping other than just putting in a TVDB id at season 1 with no episode offset. Its the ones that have an ep offset, default season != 1, or mapping-list entries that would just be missing those details for TMDB series special mappings. You might want to potentially leave those unmapped with no tsdbid or otherwise add them knowing that they are missing season/ep offset/mapping details. Looking at the currently cached "anime-list-master.xml":
So probably ~3000 entries that are not a basic 1x1 as some entries will have more than one of those flags. Thoughts on the possible options of formats of enhancing 'anime-lists' to have it have both TVDB & TMDB mapping data? Its all the same format as current, just additional tags/attributes. |
I'm personally all for it, @EndOfLine369. Regardless of TheTVDB changes, this will be an enhancement/improvement for metadata.
Side note -- does this mean the alternative ordering like Re:Zero's director's cut can be used? |
|
TVDB migration v4 agreed with API key in settings, near future but no ETA
The update engine translate the supported numbering (thetvdb, anidb) for each source, so it matches the files in Plex:
I do not see how to support more ordering modes easily. We could add tmdb support but:
https://github.com/ZeroQI/Hama.bundle/blob/master/Contents/Code/AnimeLists.py
The only mapping i could find are not usable yet
While adding 'tsdb' and anidb ids in the xml is easy to https://github.com/Anime-Lists/anime-lists, as you see from the mappings, either we redo anidb to tmdb series (let's call it tsdb) mapping, doubling the size of the file, OR we create another project (json?) to map TSDB to TVDB, which could be used by the community like anime-lists is used by Hama, filebot, etc... Not only the data type (multi-season series) would be identical between tvdb and tmdb series most of the time, but other projects could benefit (read: update it as they would also use it). only the differences would be mapped tvdbid='' |
Its "anidb<->tvdb" (current mapping) & "anidb<->tsdb" (new mapping)
Since the two "mappingList" variables are the same format, no new "def tvdb_ep" or "def anidb_ep" are required. So if we have an anidb series: And if we have a TVDB/TSDB, you can just use both functions in tandem. Another thought instead of this is to just enhance the single mappingList to make it cleaner. This would obv take more rewriting though but doable. FROM:
mappingList: {
'defaulttvdbseason': '1',
'defaulttvdbseason_a': False,
'possible_anidb3': False,
's1e1_mapped': False,
'season_map': {'12921': {'max': '1', 'min': '1'}, '14098': {'max': '2', 'min': '2'}, '14956': {'max': '3', 'min': '3'}, '15600': {'max': '4', 'min': '4'}},
'TVDB': {'s1': {'12921': '0'}, 's2': {'14098': '0'}, 's3': {'14956': '0'}, 's4': {'15600': '0'}},
'tvdbcount': 4}
TO:
mappingList: {
'name': 'Slime Taoshite 300-nen, Shiranai Uchi ni Level Max ni Nattemashita',
'studio': 'some studio'
'TVDB': {
'defaultseason': '1',
'defaultseason_a': False,
'episodeoffset': '0',
'possible_anidb3': False,
's1e1_mapped': False,
'season_map': {'15201': {'max': '1', 'min': '1'}},
'mapping': {'s1': {'15201': '0'}},
'count': 1},
'TSDB': {
'defaultseason': '1',
'defaultseason_a': False,
'episodeoffset': '0',
'possible_anidb6': False,
's1e1_mapped': False,
'season_map': {'15201': {'max': '1', 'min': '1'}},
'mapping': {'s1': {'15201': '0'}},
'count': 1}
}
We should never assume they match in any way unless stated in the mapping. If the mapping will even be setup to say they match. Better to have no meta then bad meta as its easier to spot and will prompt people to get the mapping data fixed. |
|
Better no meta than wrong one, true Seem like api key for developer don't come with required pin code So a developer need to pay for subscription to have a PIN code to do the development using TheTVDB v4 API bringing them revenue? Hell no... |
|
Am I the only one wondering about what will happen to the integrations/setups beyond just plex/hama? I'm freaking out that my setup that I just just got to work kind of the way I want will break if different approaches are taken from different projects, specifically I'm thinking in Sonarr/Radarr. Is this a general concern or am I way out of line for pointing this out? |
|
most of the big apps worked with tvdb directly to work stuff out on how to deal with v4. with a lot of them doing a loadbalancer/proxy type approach where they can control the apikey on their side (otherwise keys would get leeched and used for others). the other apps are probably going to have to do the add field and push the issue off to the end user to get a key/contribute to get access. now kodi for example years ago switched default tv scraper away from tvdb, so users also could just not use tvdb on that side.. and as more people switch to moviedb/tvmaze/whatever you will see better tv data there or requests to go that direction. now, tvdb v4 did not launch as plan (no one expected it to)... so everyone is doing the wait and see approach. -- now where does xem stand on all this, the need for xem was supposed to go away with v3 api as tvdb was supposed to add the whole flexible season listing.. but it never came. maybe with v4 xem wont really be needed there. also as more of 'xem' type stuff is added to sonarr directly, the need for xem there also diminishes. btw 'master' in xem is arbitrary as your just using it as a reference point between two entities. also master is only needed to do manual episode mapping. the entities on each end could be whatever that is in the database (tvdb/scene/anidb - tvrage/trakt are disabled). years ago there was call to add tvmaze to xem but its data was so poort i did not want to go that route as it was not adding any value. i wouldnt be opposed to go that route nowdays but there is a lot of shows in xem now and thats a lot of work to go update all those... |
|
Sounds like we have until October 1st now, per an email from today. |
Yes, thats correct. It will be October 1st, received the same email. |
|
Is the current plan just to stop supporting TVDB.....? If major library revamps are going to happen, that'd be good to know. Even some of the AniDB modes depend on TVDB for seasons. Edit: A dev key can be applied for without paying for one, per some of their comments on their Git. |
|
Need to see if Plex TVDB proxy can be used passed the deadline Thinking about it but even TMDB series, how do we do the mapping? If we lose the TVDB modes, me might as well make a separate agent for TMDB series and movies... Cannot see how to remap TheTVDB to TMDB series... |
I personally would probably be fine with just using the built-in TMDB agent, I'm prioritizing TVDB metadata over AniDB right now, so the mapping doesn't really provide much benefit right now. Edit: Now I see your point, the TMDB agent doesn't search in the movie part of the database if your anime library is based on TV Series. |
|
Also, Plex employees stated that they don't care about breaking 3rd party plug-ins: https://forums.plex.tv/t/solved-dsmv7-how-to-update-plugins-agents-scanners/700715/15 So the future doesn't look too bright for custom agents anyways. |
I would highy prefer it to be able to use TVDBv4 in the future. Just as an option for the users which are willing to pay for it. For me it would be a pain in the a.. to change all my metadata (got about 3000 Anime in my Plex). Regarding TMDB: The problem here is that there are a lot of issues with the metadata like correct seasoning, titles etc., i tried it with a few of my series and it ended in a mess. |
I'm guessing at this point, it's going to be up to folks to figure things out on their own. With TheTVDB breaking things, things from HAMA and ASS are going to break as well. Pure AniDB via HAMA or moving to Plex's TVDB are likely the future options that exist. Not ideal, but if @ZeroQI does not want to support TheTVDB given their (poor) choices, then I'm guessing it's likely the end of an era. |
|
The issue is the mapping, without AniDB to TVDB mapping due to the need for users to pay for TheTVDB access, then it will be ANiDB OR TheMovieDb series, but not mapped to each other, and this need coding, not ideal... If i spend quite some time to fix theTVDB to migrate v2 to v4 API, then TheTVDB will earn money over a user filled database, which i morally object to, and users will be paying them... Somehow i am annoyed i don't get a cut... I am not using Plex anymore since few years, reading files directly from the nas, my NAS has been shutdown for month, and i busy finishing my house in France where i work from home currently, as i am short on funds to finish, so taking a month of free time to code that is definitely not in my priority, despite the impeding doom. Still thinking of hte way forward, but i have no way to know how used are tvdb modes, apart from tvdb4 pull requests.... |
|
In theory, Anime-Lists could be updated to include TheMovieDB. TheMovieDB does have a field for External IDs which contains TheTVDB ID, so I imagine we might be able to query their database using TheTVDB ID and pull down TheMovieDB. This would create the three-way link, in theory. Ultimately tho, it's understandable how you feel about TheTVDB. I don't like it either and was hoping a solution could be found. For now, I think I'm going to start the slow and painful process of moving my library over to Plex's native systems and figure out the best way forward. |
|
My experience of the plex series agent:
I'm currently also trying out Jellyfin instead of Plex. Can't really say anything about it yet, I only installed it 2 days ago. It has plugins for AniDB and Anilist, but of course no mapping function. |
|
Movies and series is only possible in Plex with ASS to my knowledge, and if Plex cancels scanners it's gone TVDB modes is only ass/hama as well... I need to know if Plex is heading towards removing scanner and agent support as the new library/agents hints which no longer support python If so no point in developing short lived code... |
Well, nothing is said about stopping the support for third party Scanners in Plex at the moment, i could also not remember me that there was said something about this. Plex chose a way to continuesly use the TVDB API after the 1st of October this year in implementing an own Scanner and Metadata Agent directly in to Plex which is using multiple sources as source for the metadate. What about implementing a feature in Hama to let the users decide if they want to use TVDB v4 by manually entering the own API-Key and PIN into the settings of Hama->TVDB and users which not have entered a Key and PIN can use Hama without TVDB? With this you can let the users decide. |
|
Plex must be having a deal with TheTVDB unless the agent ask for an API key and pin code... Can you check? |
The TVDB agent still uses v2. The Plex TV agent is part of Plex binaries, so you'd either have to decompile the binaries or break open the HTTPS connection between Plex and TVDB. |
I think so ... otherwise I can´t explain why they are still be able to access thetvdb. As the Plex TV Series and Plex Movies Agent is a part of the Plex-Binaries i think you will not find out if they are using any API-Keys or however they are querying tvdb. |
|
Need the agent code, however it seems it won't be python anymore and could be compiled in binary, thus cannot load it unless I have a file location where it is stored... |
|
Retaining tvdb mode would be cool, so we could read or borrow TheTVDB API from Plex, since it is a Plex agent so not too diferrent from a standard tvdb library, or just access TheTVDB with no API searching the code in the web page (dirty!)... Unsure about Plex tvdb proxy, if we can use it to retain functionality by changing our URL to it in the meantime... |
|
October 1st has passed, seems like the old API is still available? |
|
correct, old api is sticking around for a bit per tvdb email last week
|
|
Is there any way to help the process to update Hama to either TVDBv4 or the migration to TMDB ? |
|
It looks like someone on reddit asked the TVDB account how Plex agents like Hama should handle v4 licensing and it sounds like TVDB will approve Hama and anything else Plex related without needing a subscription if someone contacts them. https://old.reddit.com/r/PleX/comments/iq2eud/thetvdb_user_subscriptions/hp9akop/ |
|
Also, it seems plex started to support their own database which is an aggregate from multiple databases imdb,tmdb,tvdb you can see it at https://watch.plex.tv/ , not sure if they provide support for external clients though. They also support providing the old databases GUIDs |
|
It appears that plex is taking away plugin support altogether. What will happen to hama? In case, by staying at a PMS before the plugins are removed, will support continue? Emby and Jellyfin are not on the level of hama... especially with AniDB. With hama I can use metadata from AniDB and images from other sources (which emby and jellyfin you have to manually enter the IDs and you are still limited to IDs only for TV Show). |
Any sources for this ? i haven't been able to find any official word of plugins sunsetting completely, if it's comes to that, you can always switch to NFO scanner and add your data via tiny media manager or any other source including images etc. |
|
That's has been the case since 2018' afaik any plugins that interact with frontend stopped unless you use some other kind of viewer like kitana which exposes this UI. No news on completely shutting off plugins that deal with metadata and/or scanners, so far. |
https://forums.plex.tv/t/new-plex-tv-series-agent-anime-library-pro-and-cons/798708/2
|
I see, he is not Plex employee though, I highly doubt plex will kill off metadata, scanners support anytime soon. Edit: |
|
New agents don't support python so they could do without third party agent soon. Staying on the latest version that support agents is possible but will complicate things. None of the others metadata players use python and none have custom scanners, just agents... I can help people wanting to duplicate on other platforms if they so wish but cannot see anything compelling to be honest |
I agree, do you see the possibility of making hama standalone tool that can write NFO files for use in other media servers like jellyfin/emby/kodi etc. |
|
I understand, at the moment I am testing emby and jellyfin (they are still in the process of fixing the mismatched things). In my case, the transition is atrocious. This is because I have many series such as monogatari, strike the blood, grisaia, all tenchi muyo relative in single series on collections + movies such as Fate/HF and Camelot. Thanks anyway for the reply 👍 |
|
Can Hama continue to function as stand-alone with Plex? Plex supports "Local Media Assets" but that doesn't include metadata I think. At minimum I hope that Plex can offer a route similar to jellyfin/ember/kodi so we can override Plex's agent entirely with our own data. |
|
With old style libraries it should support custom scanner and agent. New style libraries do not support python anymore and thus third party agents |
|
Will Hama be moving to TVDB API v4 soon? There are issues with the current API that TVDB won't fix because they no longer support it. For example, |
|
@ZeroQI any chance of TVDB v4 being implemented soon? |
This post is to decide the general way to go, as far as mapping file format and metadata sources go...
HAMA is currently using TheTVDB v2 API, latest being v4 [https://app.swaggerhub.com/apis/thetvdb/tvdb-api_v_4] and the only one to be supported soon
https://www.facebook.com/TheTVDB/ posted the 16th of March
https://thetvdb.com/subscribe indicate two licensing modes
— licensed - for companies, not suitable for open-source projects
— End-User Subscriptions for 11.99$ [https://thetvdb.com/subscribe] or free by filling missing IMDB|EIDR IDs [no yet available] but i could create a key as developer for free
EIDR IDs [https://www.eidr.org/] are unique IDs that look nice in principle and would have been great as unique ID but never heard of them.
As a result
https://community.synology.com/enu/forum/1/post/141423
Plex had a TheTVDB proxy i may use or my agent could share their agent key (not sure it is good) or just work with their agent (hama secondary agent or not)
ShokoAnime/ShokoServer#755 Shoko does support Plex as agent and scanner, but doubt it has HAMA flexibility for numbering and no other metadata system support movies and series in a single library (No even plex does normally)
I think TheMovieDB is the way to go since already supported, and supports both movies and series
We could keep TheTVDB with API keys for end users that subscribe, but can't really get over the fact it takes user filled data to sell back to users, without giving free keys to uploaders...
Mapping
Ideally, AniDB (Anime movies/series) and MovieDb (anime/live movies/series) mapping makes perfect sense but would require a new database format on the same format as the current Anime-List but mapped to TheMovieDB instead, but that is some undertaking...
The text was updated successfully, but these errors were encountered: