Error: [Errno 2] No such file or directory. Files exist can touch and ls. #18

DuzAwe · 2023-07-27T21:45:13Z

I am getting the bellow. Very confused as I can see the files. Running on Unraid

`root@TheLibrary:/mnt/user/scripts/PlexCache# python3 plexcache.py
*** PlexCache ***
Debug mode is active, NO FILE WILL BE MOVED.
Fetching DuzAwe's onDeck media...
Editing file paths...
Fetching subtitles...
Fetching watched media...
Fetching DuzAwe's watched media...
Editing file paths...
Fetching subtitles...
Nothing to move to array
Error: [Errno 2] No such file or directory: "/mnt/user/TV/It's Always Sunny in Philadelphia (2005) [TVdb-75805]/Season 13/It's.Always.Sunny.in.Philadelphia.-.S13E03.-.The.Gang.Beats.Boggs-.Ladies.Reboot.WEBDL-1080p.mkv"
root@TheLibrary:/mnt/user/scripts/PlexCache# nano plexcache_settings.json
root@TheLibrary:/mnt/user/scripts/PlexCache# python3 plexcache.py
*** PlexCache ***
Fetching DuzAwe's onDeck media...
Editing file paths...
Fetching subtitles...
Loading watched media from cache...
Nothing to move to array
Error: [Errno 2] No such file or directory: "/mnt/user/TV/It's Always Sunny in Philadelphia (2005) [TVdb-75805]/Season 13/It's.Always.Sunny.in.Philadelphia.-.S13E03.-.The.Gang.Beats.Boggs-.Ladies.Reboot.WEBDL-1080p.mkv"
root@TheLibrary:/mnt/user/scripts/PlexCache# ^C
root@TheLibrary:/mnt/user/scripts/PlexCache# ^C
root@TheLibrary:/mnt/user/scripts/PlexCache# ls /mnt/user/TV/It's Always Sunny in Philadelphia (2005) [TVdb-75805]/Season 13/

^C
root@TheLibrary:/mnt/user/scripts/PlexCache# python3 plexcache.py
*** PlexCache ***
Fetching DuzAwe's onDeck media...
Editing file paths...
Fetching subtitles...
Loading watched media from cache...
Nothing to move to array
Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Trek - Strange New Worlds (2022) [TVdb-382389]/Season 02/Star.Trek-.Strange.New.Worlds.-.S02E07.-.Those.Old.Scientists.WEBDL-2160p.mkv'`

bexem · 2023-07-27T23:05:08Z

Would you mind sending the log file instead?

DuzAwe · 2023-07-28T08:12:02Z

Of Course Sorry, Was late when I was messing with this.
plexcache_log_20230727_2231.log
plexcache_log_20230727_2235.log
plexcache_log_20230727_2236.log
plexcache_log_20230727_2241.log
plexcache_log_latest.log

bexem · 2023-07-28T15:07:30Z

Thank you for the logs file! I had a first look and it doesn't seem you have any misconfiguration, and the script does seem to be working with other files, correct?
I am not saying there is no issue with the script, but I can't seem to find the culprit at the moment that could explain why that specific file(s) doesn't exist for the script. The only things it comes up are permissions of those files, even though I thought the script would run as root in unraid?
Just to rule it out, would you mind checking the permissions of the file that are giving you issues? Also check the permissions of the files (possibly even of the same show) that are actually working fine. ls -l

I will be able to pay more attention to this issue at the beginning of the week, I'm currently in between night shifts and my free time (and brain power) is limited.

DuzAwe · 2023-07-28T15:16:23Z

It does not appear to be move any files

Last login: Thu Jul 27 22:09:54 2023 from 192.168.0.46
Linux 6.1.38-Unraid.
root@TheLibrary:~# cd /mnt/user/scripts/PlexCache
root@TheLibrary:/mnt/user/scripts/PlexCache# ls -l
total 4325
-rw-rw-rw- 1 root root    3307 Jul 27 21:55 README.md
-rw-rw-rw- 1 root root   64589 Jul 27 22:29 plexcache.py
-rw-rw-rw- 1 root root 3642057 Jul 27 22:32 plexcache_log_20230727_2231.log
-rw-rw-rw- 1 root root  239363 Jul 27 22:36 plexcache_log_20230727_2235.log
-rw-rw-rw- 1 root root  108179 Jul 27 22:36 plexcache_log_20230727_2236.log
-rw-rw-rw- 1 root root  104966 Jul 27 22:41 plexcache_log_20230727_2241.log
lrwxrwxrwx 1 root root      59 Jul 27 22:41 plexcache_log_latest.log -> /mnt/user/scripts/PlexCache/plexcache_log_20230727_2241.log
-rw-rw-rw- 1 root root     778 Jul 27 22:41 plexcache_settings.json
-rw-rw-rw- 1 root root   18264 Jul 27 21:55 plexcache_setup.py
-rw-rw-rw- 1 root root   85346 Jul 27 22:36 plexcache_watched_cache.json
-rw-rw-rw- 1 root root      16 Jul 27 21:55 requirements.txt
root@TheLibrary:/mnt/user/scripts/PlexCache# ls
README.md                        plexcache_log_20230727_2236.log  plexcache_setup.py
plexcache.py                     plexcache_log_20230727_2241.log  plexcache_watched_cache.json
plexcache_log_20230727_2231.log  plexcache_log_latest.log@        requirements.txt
plexcache_log_20230727_2235.log  plexcache_settings.json
root@TheLibrary:/mnt/user/scripts/PlexCache# nano plexcache_settings.json
root@TheLibrary:/mnt/user/scripts/PlexCache# cd /mnt/ingress/
root@TheLibrary:/mnt/ingress# ls -l
total 16
drwxrwxrwx 4 nobody users 4 Jul 24 14:32 Back\ Up/
drwxrwxrwx 3 nobody users 3 Jul 28 05:44 Movies/
drwxrwxrwx 9 nobody users 9 Jul 26 11:17 TV/
drwxrwxrwx 5 nobody users 5 Jul 27 21:55 scripts/
root@TheLibrary:/mnt/ingress# cd TV/
root@TheLibrary:/mnt/ingress/TV# ls -l
total 7
drwxrwxrwx 3 nobody users 3 May 21 19:07 Foundation\ (2021)\ [tvdb-366972]/
drwxrwxrwx 3 nobody users 3 Jul 28 01:24 Good\ Omens\ (2019)\ [tvdb-359569]/
drwxrwxrwx 3 nobody users 3 May 22 07:12 Harley\ Quinn\ (2019)\ [tvdb-365677]/
drwxrwxrwx 3 nobody users 3 Jul 27 21:04 Looney\ Tunes\ Cartoons\ (2020)\ [tvdb-380849]/
drwxrwxrwx 3 nobody users 3 May 21 19:08 Love,\ Death\ &\ Robots\ (2019)\ [tvdb-357888]/
drwxrwxrwx 3 nobody users 3 Jun 18 00:59 What\ We\ Do\ in\ the\ Shadows\ (2019)\ [tvdb-358211]/
drwxrwxrwx 3 nobody users 3 Jul  3 03:54 Witcher,\ The\ (2019)\ [tvdb-362696]/
root@TheLibrary:/mnt/ingress/TV#

DuzAwe · 2023-07-28T15:17:12Z

Also don't worry about it. I assume your NHS, don't be stressing about hobby bugs

bexem · 2023-07-28T15:26:34Z

Also don't worry about it. I assume your NHS, don't be stressing about hobby bugs

Guilty! I've updated the script ever so slightly:

The script will not exit while in debug and encountering an issue during file 'movement', hopefully it will make it easier to understand where the issue is.
Introduced a fixed list of subtitles extensions, as I noticed from your log it was moving the .nfo files as well as the media.

That is probably it for today.

Also...thank you so so much! ❤️

DuzAwe · 2023-07-28T16:18:00Z

Afraid, It still thinks the file doesn't exist. It's hardly as straight forward as it not escaping the ( ) or [ ].

So the behaviour is the same, It trys the first file and doesn't progress. I git pulled before I ran.

plexcache_log_20230728_1712.log
plexcache_log_latest.log

bexem · 2023-07-28T16:49:55Z

So as far I understood the file /mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 05/Star.Wars-.The.Clone.Wars.-.S05E17.-.Sabotage.Bluray-1080p.mkv does exist in your system. Is it present in the user0 folder or in the cache?

My bad about the exiting, I forgot that logging.critical also halt the script, now it should not stop according to the logic. I've also quickly added another check when calculating the free space.
It still doesn't fix the issue, but I'm trying to understand what's going on in you case.

Time to go for real this time!

DuzAwe · 2023-07-28T20:07:28Z

It would be in the user. /mnt/user/TV.

cache is a practically empty zpool can confirm the script moves nothing to it.

DuzAwe · 2023-07-28T20:20:05Z

plexcache_log_latest.log

First file fails, doesn't try the rest. All the files exist on the array under user.

bexem · 2023-07-29T14:48:43Z

I don't think I'm gonna be able to do much today or tomorrow but I'm actively thinking on how to solve this issue.

When the debug mode is off, the script is designed to exit in case it encounters an error during the moving process. I've removed this logic in this version of the file plexache.
My idea is to let the script move at least some files to see if there is any underlying issue I'm/We're not seeing and could lead to a solution.
Perhaps might be better to set the script to copy only the minimum necessary files and have a backup of the data, just in case the script accidentally corrupts any files. It is unlikely but it's always better to be safe than sorry.

I find it odd that some other users, including myself (though I may be biased), use the script without any issues. I'm thinking openly, and the only difference I can think of is the zpool. However, I cannot confirm this since I don't know the exact setup of other users, so they might actually use one already. Regardless, whether using zpool or not, it should be completely transparent to the script and shouldn't matter at all, that's why I'm not focusing on this but I just wanted to share this intrusive thought.

DuzAwe · 2023-07-29T21:03:35Z

plexcache_log_latest.log

Well-aware coding is 99% in the brain, not the computer. Ran the new file same result. To test that theory you have I've run the script with it pointing at a disk on the array rather than the cache. Still wouldn't work, however I removed the () and [] from two of the directory and they no longer trigger the error....but there does seem to be an issue with the zpool.

See attached when going to disk13 (xfs) it appears to work with an estimated move size. With the target being ingress (zpool) It tells me there are no files to move.

brackets.removed.debug.to.zpool.log
brackets.removed.debug.to.disk.log

bexem · 2023-07-30T05:05:52Z

Good morning!

Thank you for the testing! It made me think a bit. Apparently I have to properly handle zpools with the script, so I have to test if any of the sources and/or destination are zpools and act accordingly. This will take a few hours and testing, unfortunately, I will work blind on it because I have no zpool in my Unraid system. So I beg for your patience.

Regarding the parenthesis, well, now I've wrapped the folder/file name properly, so it should not give any more issues.

I will work on it between later tonight or tomorrow. Now that I have a potential solution in my mind, I'm quite excited to see how it turns out and improve the script further. I'll keep you updated on my progress!

bexem · 2023-07-31T16:01:31Z

So I've made some edits but I cannot test it so I'm not sure it will work:
The script now checks the file system of the source and destination and uses "zfs clone" and "zfs destroy" to move files around. It will also calculate the free space accordingly as zfs also handle that differently.

Please let me know if it works at all!

Forgot to say, this is the new version: zfs

DuzAwe · 2023-07-31T16:30:51Z

Ran it, No logs were produced. Still getting the file error.

Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 06/Star.Wars-.The.Clone.Wars.-.S06E01.-.The.Unknown.Bluray-1080p.mkv' Execution time of the script: 15 seconds

DuzAwe · 2023-07-31T16:35:31Z

Will "zfs clone" and "zfs destroy" work of the array is XFS?

bexem · 2023-07-31T17:06:57Z

Ran it, No logs were produced. Still getting the file error.

Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 06/Star.Wars-.The.Clone.Wars.-.S06E01.-.The.Unknown.Bluray-1080p.mkv' Execution time of the script: 15 seconds

I can't understand why the script hasn't written the log file, I did not touch that at all, and even in the event of an error during the zfs operation, it will still produce a log entry, also I can see it did end the reach as it printed the execution time, which is only invoked at the end of the script.
Just odd.
I've added a few extra logging info to locate exactly where the error happens.

Will "zfs clone" and "zfs destroy" work of the array is XFS?

The script will use the zfs clone and destroy commands only when the file system is ZFS, otherwise it will use 'normal' commands for moving files.

bexem · 2023-07-31T17:39:40Z

Before running the last updated version, please install psutil (pip install psutil) because I'm using that to check for the free space as it is more system agnostic and should work on ZFS too.

DuzAwe · 2023-08-01T08:00:35Z

Syntax error and my pythons not good enough to figure it out :P

:/mnt/user/scripts/PlexCache# python3 plexcache_zfs.py
  File "/mnt/user/scripts/PlexCache/plexcache_zfs.py", line 1144
    if check_path_exists(user_path)
                                   ^
SyntaxError: invalid syntax

bexem · 2023-08-01T18:39:42Z

Sorry if I took so long to actually answer. My bathroom decided was a good time to leak some water.

Either way I have no idea what the hell happened in my code, it was spamming me with a very generic module error even though I did test it. I've now basically rewritten the zfs logic from scratch, tested the script and it works, but as usual, I cannot confirm the zfs logic actually works.

Thank you for your patience!

DuzAwe · 2023-08-01T18:51:46Z

Absolutely no worries, I have zen practice on Mondays so I was limited in my ability to test anything.

…

On Tue 1 Aug 2023 at 19:39, Emanuele B. ***@***.***> wrote: Sorry if I took so long to actually answer. My bathroom decided was a good time to leak some water. Either way I have no idea what the hell happened in my code, it was spamming me with a very generic module error even though I did test it. I've now basically rewritten the zfs logic from scratch, tested the script and it works, but as usual, I cannot confirm the zfs logic actually works. Thank you for your patience! — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVOY6JEUZPHG2KIAYTN7GLXTFEPRANCNFSM6AAAAAA22VMGOQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

DuzAwe · 2023-08-01T20:25:28Z

Still no logs

Fetching subtitles...
Nothing to move to array
Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 06/Star.Wars-.The.Clone.Wars.-.S06E02.-.Conspiracy.Bluray-1080p.mkv'

bexem · 2023-08-01T20:38:46Z

So no log at all? Like not even interrupted?
I get having the error, but it would still start logging at the beginning way before interacting with the new zfs logic, and even if it return an error, the script will still continue as I've put no exits on purpose.

DuzAwe · 2023-08-01T20:53:45Z

No logs from either script or branch? This is new. Otherwise still chocking don't the file
verbose-output.txt

bexem · 2023-08-01T20:58:57Z

That is so odd. So I guess if you run plexcache_exclude_file.py the log will work? The only difference between the zfs one is exactly the zfs logic, otherwise is line by line identical.
If this is the case then it means the zfs logic does make the whole thing crash (but why delete the log file? That is just weird).

DuzAwe · 2023-08-01T21:43:32Z

No log from that either, nor when I switch back to main....odd there where logs. All the permissions are nobody and the scripts are root.

bexem · 2023-08-01T21:48:37Z

So you are not getting any logs at all? Have you tried setting another folder for the logs?
It's weird that it just stopped working for you, but I can't see the issue lying in the script to be fully honest with you. Not that it matters, I will keep trying to help anyway.

script_folder = "/mnt/user/system/plexcache/" # Folder path for the PlexCache script storing the settings, watchlist & watched cache files
logs_folder = script_folder # Change this if you want your logs in a different folder

DuzAwe · 2023-08-01T22:32:11Z

.
plexcache_log_latest.log
.......Looking in the wrong folder. Here are the logs.

bexem · 2023-08-01T22:46:12Z

It happens, at least is sorted!
Just to be sure that is the log for the latest zfs script? If you don't mind getting the latest version (just pushed), try it and send the log again?

DuzAwe · 2023-08-02T08:06:19Z

Most recent run from the most recent push.
plexcache_log_latest.log

bexem · 2023-08-02T17:09:27Z

Thank you for testing.
I've changed the logic to check the filesystem as it was actually giving some issue if run locally in a test script. It doesn't appear it was the issue per se judging from the log but because it's a wrapped function it might not highlight the error as it should. Worth a shot...step by step will get there.
I'm not an expert, I'm sure someone would be able to look at the code and fix it in a couple of minutes, but unfortunately I'm still a noob XD

DuzAwe · 2023-08-02T20:00:09Z

plexcache_log_latest.log
No worries, Your leagues ahead of me. I gave up the code life to be a sys admin. Still getting the error.
Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 06/Star.Wars-.The.Clone.Wars.-.S06E04.-.Orders.Bluray-1080p.mkv'

bexem · 2023-08-02T21:11:49Z

So the error seems to happen when checking for the total size of the files to move, I've swapped the function so instead of os.path.getsize() I'm using os.stat() instead that should be more system agnostic.
I've also added some extra debug in the function itself to see where exactly is getting stuck, I think we are getting closer.

DuzAwe · 2023-08-02T22:08:07Z

plexcache_log_latest.log
We have a slightly different error.
rootfs rootfs ('/mnt/ingress/Movies/The Whale (2022)/The Whale (2022) [imdb-tt13833688][Remux-2160p][DV HDR10][DTS-HD MA 7.1][HEVC]-CiNEPHiLES.mkv', '/mnt/user0/Movies/The Whale (2022)') Editing file paths... Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 06/Star.Wars-.The.Clone.Wars.-.S06E05.-.An.Old.Friend.Bluray-1080p.mkv' Execution time of the script: 1 minute, 4 seconds

bexem · 2023-08-02T22:13:10Z

So it appears to have issues when calculating the total size of the files...
I've temporarily removed it, I will fix it later, I need to know what else doesn't work.

DuzAwe · 2023-08-03T08:55:26Z

plexcache_log_latest.log

still No such file or directory:

bexem · 2023-08-03T14:41:18Z

Was this actually moved?
INFO - Moved file from /mnt/user0/Movies/The Machine (2023)/The Machine (2023) [imdb-tt11040844][AMZN][WEBDL-2160p][HDR10Plus][EAC3 5.1][h265]-CMRG.mkv to /mnt/ingress/Movies/The Machine (2023)

I suppose your /user0/ which refers to the actual array, it's not zfs, right? Not that it matter for the end goal but I want to understand why is giving errors if it isn't zfs...

DuzAwe · 2023-08-03T14:44:17Z

It would appear yes that is on Ingress, so it did move. So that's progress! \0/
The array is XFS only the pools are ZFS, Cache and Ingress.

bexem · 2023-08-03T15:01:15Z

That's good, but for what I can see it used the "normal" logic rather than the zfs one to move that file. I'm not 100% sure but judging by the fact I didn't put a logging message on that logic (all the other zfs ones have it)...
In the meanwhile I've put that logging message to be sure about it and see what I will be able to do.

Unfortunately work calls me back, I will have time again in the weekend and hopefully we'll manage a working script!

DuzAwe · 2023-08-03T15:27:43Z

plexcache_log_latest.log
No worries, New logs for the future

bexem · 2023-08-05T19:50:00Z

Good evening!
I've modified the script slightly:

I've added find_disk function to get the actual disk where the given file is located, useful with unraid as it use FUSE system;
The above function is called by get_filesystem_type which has been also modified, now has better error handling, plus windows compatibility while I was at it;
The function move_file has been slightly modified in a way that it will use the above two and theoretically use the right logic depending on the filesystem;
get_total_size_of_files function that was giving issues earlier on, I've modified it so that has better error handling, it will probably give error again, but I've made also an "alt" version of the zfs script which swaps when to call the function to fetch the actual disk on unraid, that might actually permanently fix it. But I cannot test it properly because I've made a mess with my tvseries folder (nothing to do with the script).
There is also a version with no get_total_size_of_files which also doesn't check free space at all for both versions.

Let me know whenever you can 🤞🏻

DuzAwe · 2023-08-05T20:22:54Z

plexcache_log_latest_alt_nospace.log
plexcache_log_latest_alt.log
plexcache_log_latest_norm.log
plexcache_log_latest_nospace.log
All run

bexem · 2023-08-05T20:51:11Z

So there are still issues, definitely.
I will need to figure out why is still giving errors with the free space / total size calculation, the logic is sound, I will work something out, in meantime I want to actually move stuff...but the new logic seems to not be working properly judging from your log.
I've added a couple extra logging and changed the debugging function to show me the move command in a different function, this should allow me to also see the filesystems in the log when not moving files.
It's not gonna work, this is literally for debugging.

Thank you for your patience, once again ☺️

DuzAwe · 2023-08-05T23:04:20Z

plexcache_log_latest.log

More logs

bexem · 2023-08-23T17:34:47Z

plexcache_log_latest.log

More logs

Hey there! I'm really sorry for the wait. I promise I haven't forgotten about the project or your issue in specific. Life's been a whirlwind lately with work, family, and soon, university stuff. I'll jump back in as soon as I can.
Thanks for understanding and sorry again 🙌

DuzAwe · 2023-09-07T09:25:37Z

No worries

Brramble · 2023-11-01T13:46:19Z

hey @bexem does this error stop the script from working? Im running into this issue too and nothing is moving on my server

DuzAwe mentioned this issue Sep 15, 2023

Error moving file #23

Open

Error: [Errno 2] No such file or directory. Files exist can touch and ls. #18

Error: [Errno 2] No such file or directory. Files exist can touch and ls. #18

Comments

DuzAwe commented Jul 27, 2023

bexem commented Jul 27, 2023

DuzAwe commented Jul 28, 2023

bexem commented Jul 28, 2023

DuzAwe commented Jul 28, 2023

DuzAwe commented Jul 28, 2023

bexem commented Jul 28, 2023

DuzAwe commented Jul 28, 2023 • edited Loading

bexem commented Jul 28, 2023

DuzAwe commented Jul 28, 2023

DuzAwe commented Jul 28, 2023

bexem commented Jul 29, 2023

DuzAwe commented Jul 29, 2023

bexem commented Jul 30, 2023

bexem commented Jul 31, 2023 • edited Loading

DuzAwe commented Jul 31, 2023

DuzAwe commented Jul 31, 2023

bexem commented Jul 31, 2023 • edited Loading

bexem commented Jul 31, 2023 • edited Loading

DuzAwe commented Aug 1, 2023

bexem commented Aug 1, 2023

DuzAwe commented Aug 1, 2023 via email

DuzAwe commented Aug 1, 2023

bexem commented Aug 1, 2023

DuzAwe commented Aug 1, 2023

bexem commented Aug 1, 2023

DuzAwe commented Aug 1, 2023

bexem commented Aug 1, 2023

DuzAwe commented Aug 1, 2023

bexem commented Aug 1, 2023

DuzAwe commented Aug 2, 2023

bexem commented Aug 2, 2023

DuzAwe commented Aug 2, 2023 • edited Loading

bexem commented Aug 2, 2023

DuzAwe commented Aug 2, 2023

bexem commented Aug 2, 2023

DuzAwe commented Aug 3, 2023

bexem commented Aug 3, 2023

DuzAwe commented Aug 3, 2023

bexem commented Aug 3, 2023

DuzAwe commented Aug 3, 2023

bexem commented Aug 5, 2023

DuzAwe commented Aug 5, 2023

bexem commented Aug 5, 2023

DuzAwe commented Aug 5, 2023

bexem commented Aug 23, 2023

DuzAwe commented Sep 7, 2023

Brramble commented Nov 1, 2023 • edited Loading

DuzAwe commented Jul 28, 2023 •

edited

Loading

bexem commented Jul 31, 2023 •

edited

Loading

bexem commented Jul 31, 2023 •

edited

Loading

bexem commented Jul 31, 2023 •

edited

Loading

DuzAwe commented Aug 2, 2023 •

edited

Loading

Brramble commented Nov 1, 2023 •

edited

Loading