Dt e3 0 update #304

SebastianKarpetaDev · 2024-03-15T12:47:47Z

No description provided.

keep netrc

Added preferences method implementation to User and Organization classes.

When we migrate a build history it's possible that some jobs are still running on .org. We shouldn't count them towards concurrency limits because they're technically running on .org

When we migrate build history some of the builds may be in the created state. We don't want to enqueue them as then they would be running on both platforms.

ws_config is a Hashr, which doesn't have `#key?`

*_settings is redundant.

…190) I've already attempted it to fix the problem in 01139a4, but I only fixed one code path, which is currently not used, so the changes didn't go into effect. This commit applies the changes to the other code path as well. The reason we need to do it is because when we migrate jobs from .org to .com they might still be in a state that is considered by gatekeeper. So they might be started but they're running on .org not .com, thus we shouldn't count them as running on .com. The same with new jobs - we don't want to start them as they would start on .org anyway. I decided to filter these jobs in code instead of in the database because the number of jobs that we fetch from the database is relatively small and if I had to do it in the DB I would have to always join jobs with repositories, which actually might be slower.

cherry pick commits from the public repo

This reverts commit 455b3f0.

This reverts commit e8f725d.

Allow ARM builds for Open Source only

Allow IBM power builds for Open Source only

Allow IBM z builds for Open Source only

Revert "Allow IBM z builds for Open Source only"

Stop cancelled build jobs from being queued

queue the job if there is an exception

Restrict ruby gem-update to 3.4.22 because it still supports ruby 2.6

gbarc-dt · 2024-03-19T12:54:08Z

lib/travis/scheduler/config.rb

             interval: 2,
-             limit: { public: 5, education: 1, default: 5, by_owner: {}, delegate: {} },
+             limit: { public: 93939, education: 92929, default: 91919, by_owner: {}, delegate: {} },


leave at 5,1,5 , the 93939 were tcie debug only

gbarc-dt · 2024-03-19T12:54:24Z

lib/travis/scheduler/jobs/capacity/plan.rb

@@ -17,6 +17,10 @@ def accept?(job)
            super if !on_metered_plan? || billing_allowed?(job)
          end

+          def accept?(job)


double accept?

gbarc-dt · 2024-03-19T12:55:41Z

lib/travis/scheduler/record/job.rb

    config = record&.config_json if record.respond_to?(:config_json) # TODO: remove once we've rolled over
    config ||= record&.config
    config ||= read_attribute(:config) if has_attribute?(:config)
+    config = JSON.parse(config) if config.is_a?(String)


this is already at 114

gbarc-dt · 2024-03-19T12:56:36Z

lib/travis/scheduler/record/user.rb

           end
-    return false if plan[:error] || plan['plan_name'].nil?
+    return false if plan[:error] || plan["plan_name"].nil?


gbarc-dt · 2024-03-19T12:56:50Z

lib/travis/scheduler/record/user.rb


-    plan['hybrid'] || !plan['plan_name'].include?('free')
+    plan["hybrid"] || !plan["plan_name"].include?('free')


gbarc-dt · 2024-03-19T12:57:22Z

lib/travis/scheduler/serialize/worker.rb

@@ -117,7 +117,7 @@ def repository_data
            default_branch: repo.default_branch,
            description: repo.description,
            server_type: repo.server_type || 'git'
-          )
+          server_type: repo.server_type || 'git',)


double setting and ',' (see 119)

gbarc-dt · 2024-03-19T12:57:50Z

lib/travis/scheduler/serialize/worker/job.rb

-          def vm_configs
-            config[:vm_configs] || {}
-          end
+            def vm_configs


indents broken

gbarc-dt · 2024-03-19T13:02:27Z

spec/travis/scheduler/jobs_spec.rb

@@ -1066,4 +1066,155 @@ def subscribe(plan, owner = user)
      end
    end
  end
+
+  context 'when user is on a metered plan' do


if something related to billing is on enterprise branch and isn't on master - it was removed for some reason, better don't bring it back there

gbarc-dt · 2024-03-19T13:03:08Z

spec/travis/scheduler/serialize/worker_spec.rb

        vm_config: {},
+        vm_type: :default,
+        vm_size: nil,


double vm_size

gbarc-dt · 2024-03-19T13:03:42Z

spec/travis/scheduler/service/event_spec.rb

@@ -5,7 +5,8 @@
  let(:repo)    { FactoryBot.create(:repo) }
  let(:owner)   { FactoryBot.create(:user) }
  let(:build)   { FactoryBot.create(:build, repository: repo, owner:, jobs: [job]) }
-  let(:job)     { FactoryBot.create(:job, private: true, state: :created, config: config.to_h) }
+  let(:job_stage)   { FactoryGirl.create(:stage) }


FactoryBot, there's no FactoryGirl anymore

gbarc-dt · 2024-03-19T13:03:48Z

spec/travis/scheduler/service/enqueue_owners_spec.rb

@@ -5,6 +5,8 @@
  let(:repo)    { FactoryBot.create(:repo, owner:) }
  let(:owner)   { FactoryBot.create(:user) }
  let(:commit)  { FactoryBot.create(:commit) }
+  let(:build)   { FactoryGirl.create(:build, repository: repo, owner: owner, jobs: [job]) }


FactoryBot, there's no FactoryGirl anymore

gbarc-dt · 2024-03-19T13:04:17Z

spec/travis/scheduler/service/event_spec.rb

@@ -28,7 +29,7 @@

    it { expect(log).to include 'Evaluating jobs for owner group: user svenfuchs, org travis-ci' }
    it { expect(log).to include "enqueueing job #{Job.first.id} (svenfuchs/gem-release)" }
-    it { expect(log).to include 'user svenfuchs, org travis-ci capacities: public max=5, config max=1' }
+ #   it { expect(log).to include 'user svenfuchs, org travis-ci capacities: public max=5, config max=1' }


this was debug only for tcie, can be enabled

gbarc-dt · 2024-03-19T13:04:51Z

spec/travis/scheduler/service/event_spec.rb

@@ -45,4 +46,24 @@
      expect(log).to include "I 1234 Owner group scheduler.owners-svenfuchs is locked and already being evaluated. Dropping event build:created for build=#{build.id}"
    }
  end
+


for some reason removed on master, don't add it (also factorygirl used)

gbarc-dt · 2024-03-19T13:05:01Z

spec/travis/scheduler/service/notify_spec.rb

@@ -1,7 +1,9 @@
 # frozen_string_literal: false

 describe Travis::Scheduler::Service::Notify do
-  let(:job)     { FactoryBot.create(:job, state: :queued, queued_at: Time.parse('2016-01-01T10:30:00Z'), config: {}) }
+  let(:build)   { FactoryGirl.create(:build, repository: repo, owner: owner, jobs: [job]) }


factorygirl -> factorybot

factorygirl is still here for 'build'

gbarc-dt

found some factorygirl calls - pls grep the sources and replace all with factorybot

gbarc-dt · 2024-03-26T07:13:23Z

spec/travis/scheduler/service/notify_spec.rb

@@ -1,7 +1,9 @@
 # frozen_string_literal: false

 describe Travis::Scheduler::Service::Notify do
-  let(:job)     { FactoryBot.create(:job, state: :queued, queued_at: Time.parse('2016-01-01T10:30:00Z'), config: {}) }
+  let(:build)   { FactoryGirl.create(:build, repository: repo, owner: owner, jobs: [job]) }


factorygirl is still here for 'build'

gbarc-dt · 2024-03-26T07:13:53Z

spec/travis/scheduler/service/enqueue_owners_spec.rb

@@ -5,6 +5,8 @@
  let(:repo)    { FactoryBot.create(:repo, owner:) }
  let(:owner)   { FactoryBot.create(:user) }
  let(:commit)  { FactoryBot.create(:commit) }
+  let(:build)   { FactoryBot.create(:build, repository: repo, owner: owner, jobs: [job]) }
+  let(:job_stage)   { FactoryGirl.create(:stage) }


factorygirl for 'job_stage' - change to factorybot

gbarc-dt · 2024-03-26T07:14:14Z

spec/travis/scheduler/serialize/worker/job_spec.rb

+      end
+
+      context 'when head repo is present' do
+        let(:head_repo) { FactoryGirl.create(:repository, github_id: 549744) }


factorygirl - change to factorybot

gbarc-dt · 2024-03-26T07:14:38Z

spec/travis/scheduler/record/user_spec.rb

@@ -4,6 +4,8 @@
  let(:user) { FactoryBot.create(:user) }
  let(:repo) { FactoryBot.create(:repository) }
  let(:authorize_build_url) { "http://localhost:9292/users/#{user.id}/plan" }
+  let(:repo) { FactoryGirl.create(:repository) }


factorygirl - change to factorybot

GbArc · 2024-03-28T07:46:22Z

lib/travis/scheduler/service/set_queue.rb

+          rescue => e
+            puts "ERROR while trying to queue: #{e.message}"
+            puts "Backtrace:"
+            puts e.backtrace.join("\n")@queue ||= redirect(Queue.new(job, config, logger).select)


newline missing before @Queue

svenfuchs and others added 30 commits March 13, 2019 17:41

pass :keep_netrc preference to the worker

fab7c08

Merge pull request #3 from travis-pro/sf-keep_netrc

f2e72af

keep netrc

Added preferences method

4a382f9

Added preferences method to User class too.

248945a

Merge pull request #4 from travis-pro/dj-netrc-fix

dc4a2be

Added preferences method implementation to User and Organization classes.

use config import for db setup

455b3f0

Update .travis.yml

e8f725d

Don't include migrated jobs when calculating concurrency

f106d61

When we migrate a build history it's possible that some jobs are still running on .org. We shouldn't count them towards concurrency limits because they're technically running on .org

Don't enqueue jobs that are migrated from .org

f88746a

When we migrate build history some of the builds may be in the created state. We don't want to enqueue them as then they would be running on both platforms.

Put workspace_settings on payload

6d765e0

Fix workspace_settings

ed7692e

ws_config is a Hashr, which doesn't have `#key?`

Environment variables can be restricted to specified branch

4a1c644

Env model extended

e9ffa09

Tests fixes

c2c73c8

Return type improvement

88f4609

Test fixed

2936ca8

Test fix

0b795d7

Rename workspace_settings to workspace

e51a0b0

*_settings is redundant.

allow env vars to be hashes

f071b95

do not decrypt values that are not a String or Hash

6f0b2a1

Merge pull request #5 from travis-pro/sf-private

b5baad3

cherry pick commits from the public repo

Revert "use config import for db setup"

4e2bad2

This reverts commit 455b3f0.

Revert "Update .travis.yml"

059fc53

This reverts commit e8f725d.

Merge pull request #196 from travis-ci/pd-oss-only-arm

219e895

Allow ARM builds for Open Source only

Merge pull request #197 from travis-ci/pd-ibm-oss-only

41f93e4

Allow IBM power builds for Open Source only

Allow IBM z builds for Open Source only

899bd02

Merge pull request #8 from travis-pro/pd-limit-z-queues

b879509

Allow IBM z builds for Open Source only

Revert "Allow IBM z builds for Open Source only"

5f0a570

Merge pull request #9 from travis-pro/revert-8-pd-limit-z-queues

11b9b24

Revert "Allow IBM z builds for Open Source only"

GbArc and others added 6 commits May 26, 2023 14:01

Using proper key for forked secrets (#296)

ec52f0a

Stop jobs from being queued if the build has been cancelled (#298)

79d54c1

Stop cancelled build jobs from being queued

Stages are not always used ship:docker (#299)

7c7c654

Stop build from not getting queued (#300)

da76215

queue the job if there is an exception

Restrict ruby gem-update version (#302)

d16a4b4

Restrict ruby gem-update to 3.4.22 because it still supports ruby 2.6

Scheduler merge. Enterprise to master.

ba23d18

SebastianKarpetaDev requested a review from GbArc March 15, 2024 13:22

gbarc-dt reviewed Mar 19, 2024

View reviewed changes

TravisCI Scheduler merge. Enterprise to master.

1d6f840

gbarc-dt requested changes Mar 26, 2024

View reviewed changes

TravisCI Scheduler merge. Enterprise to master.

1044296

GbArc requested changes Mar 28, 2024

View reviewed changes

Sebastian Karpeta and others added 5 commits April 2, 2024 10:22

TravisCI Scheduler. New line issue.

846ae7f

billing get corrected, job save removed

cd7bd69

dist updated to focal / ship:docker

353bc62

removed duplicates from dockerfile, moved exec-hooks / ship:docker

d3ce4e9

dockerfile split ship:docker

8cc1bfd


		plan['hybrid'] \|\| !plan['plan_name'].include?('free')
		plan["hybrid"] \|\| !plan["plan_name"].include?('free')

Dt e3 0 update #304

Are you sure you want to change the base?

Dt e3 0 update #304

Conversation

SebastianKarpetaDev commented Mar 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gbarc-dt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment