Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flapping and dependant processes #151

Open
AntonMalyshev opened this issue Sep 4, 2015 · 2 comments
Open

Flapping and dependant processes #151

AntonMalyshev opened this issue Sep 4, 2015 · 2 comments
Labels
Milestone

Comments

@AntonMalyshev
Copy link

Hello! Could you advice how to setup "flapping" of process to delay its startup for 1 hour if it was crashed 10+ times in 1 minute? Also it seems that flapping settings are not applied if the process is starting as a dependency of another process. How can we fix it?

@kostya
Copy link
Owner

kostya commented Sep 4, 2015

  1. trigger :flapping, :times => 10, :within => 1.minute, :retry_in => 1.hour ?
  2. show config example where flapping not working with dependency.

@AntonMalyshev
Copy link
Author

  1. Thank you!
  2. The sample config:
Eye.application "sample/applicationname" do
  stop_on_delete true
  group 'groupname' do
      process :process1 do
          self.daemonize true
          self.pid_file "/tmp/process1.pid"
          self.start_command "process1"
          trigger :flapping, :times => 10, :within => 1.minute, :retry_in => 1.hour
      end
      process :process2 do
          self.daemonize true
          self.pid_file "/tmp/process2.pid"
          self.start_command "process2"
          depend_on :process1
          trigger :flapping, :times => 10, :within => 1.minute, :retry_in => 1.hour
      end
  end
end

Process1 is constantly crashing and after 10 crashes flapping is triggered, eye info returns:

sample/applicationname     
  groupname                        
    process1 ...................... unmonitored (flapping at 07 Sep 16:37)
    process2 ...................... unmonitored (monitor by user at 07 Sep 16:37)

But in a minute dependency of process2 is triggered and eye tries to start process1 again:

sample/applicationname     
  groupname                        
    process1 ...................... starting
    process2 ...................... starting

Log:

07.09.2015 16:37:54 ERROR -- [sample/applicationname:groupname:process1] NOTIFY: flapping!
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] schedule :unmonitor (reason: flapping)
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] schedule :check_crash (reason: crashed)
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] <= restore
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] => check_crash  (reason: crashed)
07.09.2015 16:37:54 WARN  -- [sample/applicationname:groupname:process1] check crashed: process is down
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] schedule :restore (reason: crashed)
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] <= check_crash
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] => unmonitor  (reason: flapping)
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] switch :unmonitoring [:down => :unmonitored] (reason: flapping)
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] <= unmonitor
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] => check_crash  (reason: crashed)
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] <= check_crash
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] => restore  (reason: crashed)
07.09.2015 16:37:54 INFO  -- [sample/applicationname:groupname:process1] <= restore
07.09.2015 16:38:39 INFO  -- [sample/applicationname:groupname:process2] schedule :start (reason: wait_dependency)
07.09.2015 16:38:39 INFO  -- [sample/applicationname:groupname:process2] => start  (reason: wait_dependency)
07.09.2015 16:38:39 INFO  -- [sample/applicationname:groupname:process2] pid_file not found, starting...
07.09.2015 16:38:39 INFO  -- [sample/applicationname:groupname:process2] switch :starting [:unmonitored => :starting] (reason: wait_dependency)
07.09.2015 16:38:39 INFO  -- [sample/applicationname:groupname:process1] schedule :start (reason: start_dependency)
07.09.2015 16:38:39 INFO  -- [sample/applicationname:groupname:process1] => start  (reason: start_dependency)
07.09.2015 16:38:39 INFO  -- [sample/applicationname:groupname:process1] pid_file not found, starting...
07.09.2015 16:38:39 INFO  -- [sample/applicationname:groupname:process1] switch :starting [:unmonitored => :starting] (reason: start_dependency)```

@kostya kostya added the bug label Sep 14, 2015
@kostya kostya modified the milestones: 0.7, 0.7.1 Sep 16, 2015
@kostya kostya modified the milestones: 0.8, 0.7.1 Oct 3, 2015
@kostya kostya modified the milestones: 0.9, 0.8 Nov 22, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants