Check if Include folders/files do exists (in case they are removed) #1718

rafaelhdr · 2024-05-09T10:00:48Z

Solution b) when start taking a snapshot the include list should be checked of existence first and warn about missings.

reference: #1586

Solution b) when start taking a snapshot the include list should be checked of existence first and warn about missings. reference: bit-team#1586

rafaelhdr · 2024-05-09T10:03:08Z

This is not complete yet, but if possible I would like some feedback before finishing it.

Are the messages good? And I am wondering if the qt warning is ok (it is asking a confirmation before continuing instead of just warning).

And I didn't implement a) for no big reason. I am still getting familiar with the code and b) was something that was more clear for me. I could try a) in a future PR.

And how you normally do translations? I was planning to translate all other languages using some automated tool (Copilot) but I wanted to confirm how you guys do it normally...

CHANGES

common/backintime.py

common/po/pt_BR.po

common/snapshots.py

qt/app.py

buhtz · 2024-05-09T13:45:36Z

Dear Rafael,
Awesome! Thank you for this next contribution.

I added some comments and suggestions to the code. I am not sure about the whole solution. I would suggest to further discuss it in the related issue #1586.

I will set the PR into Draft mode to state the the solution is not finished.

Best,
Christian

Co-authored-by: buhtz <[email protected]>

rafaelhdr · 2024-05-09T20:31:17Z

I appreciate a lot for the review. It clarified a lot of questions and taught me some good lessons (I would never have thought about RTL issues on translations). And I am happy about not use the camel case :)

I will do the fixes from the review and wait for any suggestion in the original issue.

Thank you very much 🙏

refactor btn snapshot & improve translate message Co-authored-by: buhtz <[email protected]>

ref: https://peps.python.org/pep-0008/#blank-lines

buhtz · 2024-05-10T11:30:22Z

Does anyone know how to trigger the systray-icon message-bubbles via BIT?

I tried with "Snapshot.setTakeSnapshotMessage(1, 'FOO BAR')" but without success. This message appears in the status bar of the main window.

buhtz · 2024-05-12T07:09:20Z

I would never have thought about RTL issues on translations

Me neither. When I started to work on the translation task I thougt it is easy. I learned things like that from translators and the community around them while trying to attract translators to BIT.

rafaelhdr · 2024-05-13T19:12:50Z

I am still checking on Systray. TBH I didn't know we had a systray (it is not running fine here). But I will keep trying to understand it in order to make the fix.

aryoda · 2024-05-13T19:41:04Z

Does anyone know how to trigger the systray-icon message-bubbles via BIT?

Our systray icon started as stand-alone process

backintime/qt/plugins/systrayiconplugin.py

Lines 86 to 93 in 99081af

    
           def processBegin(self): 
        
               try: 
        
                   logger.debug("Trying to start systray icon sub process...") 
        
                   path = os.path.join(tools.backintimePath('qt'), 'qtsystrayicon.py') 
        
                   cmd = [sys.executable, path, self.snapshots.config.currentProfile()] 
        
                   if logger.DEBUG: 
        
                       cmd.append("--debug")  # HACK to propagate DEBUG logging level to sub process 
        
                   self.process = subprocess.Popen(cmd)

and it calls every second a function to update the status:

backintime/qt/qtsystrayicon.py

Lines 160 to 201 in 99081af

    
           def updateInfo(self): 
        
               # Exit this systray icon "app" when the snapshots is taken 
        
               if not self.snapshots.busy(): 
        
                   self.prepareExit() 
        
                   self.qapp.exit(0) 
        
                   return 
        
               paused = tools.processPaused(self.snapshots.pid()) 
        
               self.btnPause.setVisible(not paused) 
        
               self.btnResume.setVisible(paused) 
        
               message = self.snapshots.takeSnapshotMessage() 
        
               if message is None and self.last_message is None: 
        
                   message = (0, _('Working…')) 
        
               if not message is None: 
        
                   if message != self.last_message: 
        
                       self.last_message = message 
        
                       if self.decode: 
        
                           message = (message[0], self.decode.log(message[1])) 
        
                       self.menuStatusMessage.setText('\n'.join(textwrap.wrap(message[1], \ 
        
                                                                                       width = 80) \ 
        
                                                                )) 
        
                       self.status_icon.setToolTip(message[1]) 
        
               pg = progress.ProgressFile(self.config) 
        
               if pg.fileReadable(): 
        
                   pg.load() 
        
                   percent = pg.intValue('percent') 
        
                   ## disable progressbar in icon until BiT has it's own icon 
        
                   ## fixes bug #902 
        
                   # if percent != self.progressBar.value(): 
        
                   #     self.progressBar.setValue(percent) 
        
                   #     self.progressBar.render(self.pixmap, sourceRegion = QRegion(0, -14, 24, 6), flags = QWidget.RenderFlags(QWidget.DrawChildren)) 
        
                   #     self.status_icon.setIcon(QIcon(self.pixmap)) 
        
                   self.menuProgress.setText(' | '.join(self.getMenuProgress(pg))) 
        
                   self.menuProgress.setVisible(True) 
        
               else: 
        
                   # self.status_icon.setIcon(self.icon.BIT_LOGO) 
        
                   self.menuProgress.setVisible(False)

It reads the current "snapshot message" from a file:

backintime/common/snapshots.py

Line 96 in 99081af

def takeSnapshotMessage(self):

So to change the status message of the systray icon the message file's content must be changed, I guess (= not tried) with:

backintime/common/snapshots.py

Line 151 in 99081af

def setTakeSnapshotMessage(self, type_id, message, timeout=-1):

I tried with "Snapshot.setTakeSnapshotMessage(1, 'FOO BAR')" but without success. This message appears in the status bar of the main window.

The message is only shown while a snapshot is taken otherwise the systray icon is closed again (it is not running permanently - which is what I would implement in the future to also get rid of the root systray icon issues due to hijacking the X11 session of a user).

aryoda · 2024-05-13T21:50:00Z

@rafaelhdr Thanks a lot for your contribution and helping us to improve BiT!

@rafaelhdr @buhtz I am wondering if the validation logic to check for existing includes does really work in every scenario, esp. when the include path is mounted via a user-callback the validation is only possible (valid ;-) after the user-callback is called.

I know this a quite rare scenario (using a user-callback as well as mounting the include folder - normally I mount the target folder - not the includes root folder) but we should test this before merging the PR.

For a simple user-callback example see https://github.com/bit-team/user-callback/blob/master/user-callback.diagnostics

The BiT function to mount for using a profile is this:

backintime/common/pluginmanager.py

Line 165 in 99081af

def mount(self, profileID = None):

buhtz · 2024-05-14T05:54:31Z

I do see. This is a longer story.

aryoda · 2024-05-14T06:58:30Z

BTW: I think to correctly test the mount/unmount user-callback calls via the BiT GUI my old patch must be applied:

#724 (comment)

Perhaps it is time to push this patch now as work-around (even though it is in-efficient because it mounts/unmounts far too often)...

buhtz · 2024-05-14T15:27:34Z

@rafaelhdr @buhtz I am wondering if the validation logic to check for existing includes does really work in every scenario, esp. when the include path is mounted via a user-callback the validation is only possible (valid ;-) after the user-callback is called.

You mean if the "backup source" is an external drive for example and present all the time?
So the check should be done after the user-callback script was executed?

aryoda · 2024-05-14T16:39:03Z

@rafaelhdr @buhtz I am wondering if the validation logic to check for existing includes does really work in every scenario, esp. when the include path is mounted via a user-callback the validation is only possible (valid ;-) after the user-callback is called.

You mean if the "backup source" is an external drive for example and present all the time? So the check should be done after the user-callback script was executed?

Yes, check for existing include folders after the user-callback script was called to "mount" everything.
An external drive would be one scenario but more realistically a network drive (if the source is not permanently mounted but only mounted for BiT and after the backup unmounted again).

Or another example: The source is an encrypted container (TrueCrypt/VeraCrypt) and shall be mounted only for backups

rafaelhdr · 2024-05-15T19:42:08Z

Thank you for all your help, guys 🙏

About the Systray, I think I got the idea. I was able to run it locally, and I will try to add something.Thank you for the directions.

BTW, I just noticed the systray after you guys mentioned it. I was able to make it work after adding an extension to Gnome (https://extensions.gnome.org/extension/615/appindicator-support/). Do you think it is worth mentioning in the docs?

About the user-callback suggestion, I think I got it... Just to be sure, these are the logs from my local tests:

$ backintime backup --profile-id 4

> Back In Time
> Version: 1.4.4-dev.7472f54d

(...)

> WARNING: The following folders are missing: /home/rafaelhdr/Mocks2
> INFO: Lock
> INFO: Mountpoint /home/rafaelhdr/.local/share/backintime/mnt/7AE1B8E6/mountpoint is already mounted
> INFO: Take a new snapshot. Profile: 4 khadas 2
(...)

So, this WARNING: The following (...) should be happening after the INFO: Mountpoint /home/(...), right? With that, we could be sure it is properly mounted...

rafaelhdr · 2024-05-20T15:58:05Z

Hey guys,

Thank you for all the support. I was able to display the error with this:

We could consider it as an error, right? (if not, I could replace the 1 here self.setTakeSnapshotMessage(1, msg))

And I moved the check for after the mount. I hope it is good.

Let me know if you think we could change it, but also feel free to change anything :)

aryoda · 2024-05-20T23:51:05Z

FYI: I am AFK for one week and will review this PR when I am back...

buhtz · 2024-06-28T07:00:39Z

Hello Rafael,
there where merge conflicts because of other PRs I merged. I solved that conflicts. So you need to update (fetch & pull) your local git repository before working further.

rafaelhdr · 2024-07-08T17:39:58Z

@buhtz thank you

@aryoda could you review this Pull Request, pls 🙏 . I am very new to the project and your eyes on it would be appreciated.

buhtz · 2024-07-08T18:55:19Z

Copy & Paste ...

Using @username to directly mention a user on GitHub should be avoided for several reasons.

Firstly, excessive @mentions can create unnecessary notifications, disturbing the users. Secondly, it can be seen as impolite to mention someone directly without prior communication or context. Thirdly, it's important to ensure that mentions are relevant and contribute to the discussion, such as when asking direct questions or involving someone in a specific part of the conversation. Microsoft GitHub recommends using mentions sparingly and thoughtfully to maintain effective communication.

Additionally, it's often unnecessary to mention admins or moderators directly, as they are usually subscribed to relevant issues or pull requests and will receive notifications automatically.

rafaelhdr · 2024-07-11T20:59:25Z

I apologize about the mention :(

aryoda · 2024-07-13T08:17:52Z

@aryoda could you review this Pull Request, pls 🙏 . I am very new to the project and your eyes on it would be appreciated.

I had no time for longer than expected but I plan to do the review tomorrow (sorry for not informing you earlier!).
I really appreciate your commitment 👍

aryoda

@rafaelhdr I have done the code review with some (more or less cosmetic) findings only. Could you please check my review findings and pull the changes (or discuss different solutions here)? THX a lot :-)

PS: I will do some manual tests on my own then using my personal testing profiles...

aryoda · 2024-05-13T20:55:57Z

common/backintime.py

@@ -48,6 +48,16 @@

 parsers = {}

+
+def warning_on_take_snapshot(config):


I am not sure it this function should be placed into snapshots.py or instead (or some other code file that is more appropriate) since backintime.py` is the "main" code file and already quite crowded.

Also: I would prefer a name like eg. validate_profile() or validate_on_take_snapshot() because this is the major purpose of the function (issuing a warning is "just" a final action and does not always happen).

Finally I am wondering if this logic could be implemented into

backintime/common/config.py

Lines 338 to 341 in 99081af

def checkConfig(self):

profiles = self.profiles()

for profile_id in profiles:

where it naturally fits but that function is doing to much at the moment (checks all profiles/configs so it is a mis-nomer - should be checkConfig() and call a separate checkProfile() for each single profile of a configuration so that the logic of warning_on_take_snapshots() could be added there and called as done below...

OK, I have checked the implementation and think using the config.py#checkConfig() is too complicated at the moment (requires some refactorings and better documentation)...

aryoda · 2024-05-13T21:33:44Z

qt/app.py

@@ -1182,12 +1182,27 @@ def updateTimeLine(self, refreshSnapshotsList = True):
                item = self.timeLine.addSnapshot(sid)
            self.timeLine.checkSelection()

+    def validate_on_take_snapshot(self):


excellent function name 👍 !!!

aryoda · 2024-05-13T21:38:28Z

common/snapshots.py

@@ -3109,6 +3109,25 @@ def lastSnapshot(cfg):
        return sids[0]


+def has_missing(included):


I would add what is missing to make the purpose of the function clear: has_missing_includes()

So I suggest to rename the function to has_missing_includes()

aryoda · 2024-07-14T14:50:19Z

common/snapshots.py

+
+        if missing:
+            msg = ', '.join(missing)
+            msg = f'The following folders are missing: {msg}'


I think "includes" could be folders but also files so I would add "... following **files/**folders are missing..." to the message

aryoda · 2024-07-14T14:53:09Z

common/snapshots.py

+            msg = ', '.join(missing)
+            msg = f'The following folders are missing: {msg}'
+            logger.warning(msg)
+            self.setTakeSnapshotMessage(1, msg)


"1" means "error" ("0" would be "info"), but we do not yet have a value for "warnings".
I am not quite sure about the impact of using "error" here (eg. does this mean that the taken snapshot is reported as "error"?). I have to test this...

Sure. I think I used 1 in order to show properly in the Systray (but now systray is not working fine for me anymore :/)

But sure. Let me know if you want me to change :)

aryoda · 2024-07-14T14:53:55Z

common/snapshots.py

@@ -698,6 +698,15 @@ def remove(self, sid):

            return True

+    def _warning_on_take_snapshot(self, config):


It would be great to add a comment header to describe the purpose and arguments of the function

aryoda · 2024-07-14T15:00:27Z

common/snapshots.py

@@ -698,6 +698,15 @@ def remove(self, sid):

            return True

+    def _warning_on_take_snapshot(self, config):


This function name (_warning_on_take_snapshot) does indicate that a warning is issued but it is unclear under which condition (or what is checked).

I suggest to rename the function to eg. _check_included_sources_exist_on_take_snapshot (perhaps you find a better word than "sources")

aryoda · 2024-07-14T15:06:14Z

common/snapshots.py

+    """
+    not_found = []
+    for path, info in included:
+        if not os.path.exists(path):


I am not sure if os.path.exists() does also work for files (though folders are files), could you please (manually) check this with a configuration that contains a "missing" and an "existing" file (positive and negative test)?

Sure. I just did it and it worked fine :)

I tested via CLI:

➜ /tmp cd /tmp /0.0s ➜ /tmp touch test.txt /0.0s ➜ /tmp python Python 3.12.4 (main, Jun 7 2024, 06:33:07) [GCC 14.1.1 20240522] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.path.exists("/tmp") True >>> os.path.exists("/tmp/test.txt") True >>> os.path.exists("/wrong") False >>> os.path.exists("/tmp/wrong.txt") False

I tested w/ CLI, and got correct results w/ this warning:

WARNING: The following **files/**folders are missing: /home/rafaelhdr/RMFolder, /home/rafaelhdr/RMFIle

And also via GUI with real snapshots (w/ existent folder, removed folder, existent file and removed file).

aryoda · 2024-07-14T15:11:17Z

qt/app.py

+        missing = snapshots.has_missing(self.config.include())
+        if missing:
+            msg_missing = '\n'.join(missing)
+            msg = _('The following folders are missing: {folders} Do you want to proceed?'.format(


The def _warning_on_take_snapshot() in snapshots.py also has a similar message creation logic so this code partially looks like duplicated code (and "includes" could also be files so I would add "files/folders` here too.

If you'd find a simple way to de-duplicate the code it would be great but if it is too complicated I could live with the existing code here.

About this, a similar code was made with the missing = snapshots.has_missing(self.config.include()) (which is now renamed to has_missing_includes)

Only the text generation is different (also not translated the CLI version) and how we warn the user.

Ok?

rafaelhdr

Thank you very much for the review. I made the improvements suggested and answered/tested per requests.

rafaelhdr · 2024-07-26T08:12:33Z

common/snapshots.py

+    """
+    not_found = []
+    for path, info in included:
+        if not os.path.exists(path):


Sure. I just did it and it worked fine :)

I tested via CLI:

➜ /tmp cd /tmp /0.0s ➜ /tmp touch test.txt /0.0s ➜ /tmp python Python 3.12.4 (main, Jun 7 2024, 06:33:07) [GCC 14.1.1 20240522] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.path.exists("/tmp") True >>> os.path.exists("/tmp/test.txt") True >>> os.path.exists("/wrong") False >>> os.path.exists("/tmp/wrong.txt") False

I tested w/ CLI, and got correct results w/ this warning:

WARNING: The following **files/**folders are missing: /home/rafaelhdr/RMFolder, /home/rafaelhdr/RMFIle

And also via GUI with real snapshots (w/ existent folder, removed folder, existent file and removed file).

rafaelhdr · 2024-07-26T08:15:22Z

qt/app.py

+        missing = snapshots.has_missing(self.config.include())
+        if missing:
+            msg_missing = '\n'.join(missing)
+            msg = _('The following folders are missing: {folders} Do you want to proceed?'.format(


About this, a similar code was made with the missing = snapshots.has_missing(self.config.include()) (which is now renamed to has_missing_includes)

Only the text generation is different (also not translated the CLI version) and how we warn the user.

Ok?

rafaelhdr · 2024-07-26T08:26:50Z

common/snapshots.py

+            msg = ', '.join(missing)
+            msg = f'The following folders are missing: {msg}'
+            logger.warning(msg)
+            self.setTakeSnapshotMessage(1, msg)


Sure. I think I used 1 in order to show properly in the Systray (but now systray is not working fine for me anymore :/)

But sure. Let me know if you want me to change :)

Check if Include folders/files do exists (in case they are removed)

f6bf5bc

Solution b) when start taking a snapshot the include list should be checked of existence first and warn about missings. reference: bit-team#1586