Parallel optimizer #247

kasyanovse · 2023-12-07T13:19:46Z

WIP

Продолжение работы #199.

Этап 1. Воссоздание существующего генетического оптимизатора с использованием нового подхода

Этап 2. Добавление бандитов для мутаций

Переписать бандитов под новый оптимизатор
Добавить бандитов в новый оптимизатор

pep8speaks · 2023-12-07T13:19:57Z

Hello @kasyanovse! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file golem/core/optimisers/common_optimizer/common_optimizer.py:

Line 1:1: F401 'abc.abstractmethod' imported but unused
Line 2:1: F401 'collections.deque' imported but unused
Line 3:1: F401 'dataclasses.dataclass' imported but unused
Line 4:1: F401 'multiprocessing.managers.DictProxy' imported but unused
Line 5:1: F401 'typing.Dict' imported but unused
Line 5:1: F401 'typing.List' imported but unused
Line 5:1: F401 'typing.Callable' imported but unused
Line 10:1: F401 'golem.core.optimisers.common_optimizer.node.Node' imported but unused
Line 13:1: F401 'golem.core.optimisers.common_optimizer.scheme.Scheme' imported but unused
Line 14:1: F401 'golem.core.optimisers.common_optimizer.stage.Stage' imported but unused
Line 15:1: F401 'golem.core.optimisers.common_optimizer.task.Task' imported but unused
Line 15:1: F401 'golem.core.optimisers.common_optimizer.task.TaskStatusEnum' imported but unused
Line 17:1: F401 'golem.core.optimisers.graph.OptGraph' imported but unused
Line 18:1: F401 'golem.core.optimisers.objective.ObjectiveFunction' imported but unused
Line 20:1: F401 'golem.core.optimisers.opt_history_objects.opt_history.OptHistory' imported but unused
Line 22:1: F401 'golem.core.optimisers.optimizer.GraphOptimizer' imported but unused
Line 24:1: F401 'golem.core.optimisers.timer.OptimisationTimer' imported but unused

In the file golem/core/optimisers/common_optimizer/common_optimizer_params.py:

Line 1:1: F401 'abc.abstractmethod' imported but unused
Line 2:1: F401 'collections.deque' imported but unused
Line 4:1: F401 'typing.Callable' imported but unused
Line 7:1: F401 'golem.core.optimisers.common_optimizer.node.Node' imported but unused
Line 8:1: F401 'golem.core.optimisers.common_optimizer.old_config.default_stages' imported but unused
Line 9:1: F401 'golem.core.optimisers.common_optimizer.scheme.Scheme' imported but unused
Line 10:1: F401 'golem.core.optimisers.common_optimizer.stage.Stage' imported but unused
Line 11:1: F401 'golem.core.optimisers.common_optimizer.task.Task' imported but unused
Line 11:1: F401 'golem.core.optimisers.common_optimizer.task.TaskStatusEnum' imported but unused
Line 12:1: F401 'golem.core.optimisers.genetic.operators.operator.EvaluationOperator' imported but unused
Line 13:1: F401 'golem.core.optimisers.graph.OptGraph' imported but unused
Line 14:1: F401 'golem.core.optimisers.objective.ObjectiveFunction' imported but unused
Line 15:1: F401 'golem.core.optimisers.opt_history_objects.individual.Individual' imported but unused
Line 18:1: F401 'golem.core.optimisers.optimizer.GraphOptimizer' imported but unused
Line 19:1: F401 'golem.core.optimisers.populational_optimizer.PopulationalOptimizer' imported but unused
Line 20:1: F401 'golem.core.optimisers.timer.OptimisationTimer' imported but unused
Line 39:1: W391 blank line at end of file

In the file golem/core/optimisers/common_optimizer/config.py:

Line 3:1: F401 'copy.deepcopy' imported but unused
Line 4:1: F401 'enum.Enum' imported but unused
Line 4:1: F401 'enum.auto' imported but unused
Line 8:1: F401 'numpy as np' imported but unused
Line 9:1: F401 'typing.Optional' imported but unused
Line 9:1: F401 'typing.Tuple' imported but unused
Line 9:1: F401 'typing.List' imported but unused
Line 31:1: E303 too many blank lines (3)
Line 40:1: E302 expected 2 blank lines, found 1
Line 49:1: E302 expected 2 blank lines, found 1
Line 52:1: E305 expected 2 blank lines after class or function definition, found 1
Line 75:1: E302 expected 2 blank lines, found 1
Line 86:1: E302 expected 2 blank lines, found 1
Line 98:1: E305 expected 2 blank lines after class or function definition, found 1
Line 114:1: E302 expected 2 blank lines, found 0
Line 118:1: E305 expected 2 blank lines after class or function definition, found 1
Line 125:1: E303 too many blank lines (3)
Line 149:1: W391 blank line at end of file

In the file golem/core/optimisers/common_optimizer/nodes/crossover.py:

Line 11:36: F821 undefined name 'CommonOptimizerParameters'
Line 18:45: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/nodes/evaluator.py:

Line 3:1: F401 'golem.core.optimisers.genetic.operators.mutation.Mutation as OldMutation' imported but unused
Line 7:36: F821 undefined name 'CommonOptimizerParameters'
Line 12:45: F821 undefined name 'CommonOptimizerParameters'
Line 15:1: E305 expected 2 blank lines after class or function definition, found 1
Line 15:1: E402 module level import not at top of file
Line 15:1: F401 'golem.core.optimisers.fitness.null_fitness' imported but unused
Line 16:1: E302 expected 2 blank lines, found 0

In the file golem/core/optimisers/common_optimizer/nodes/mutation.py:

Line 7:1: F401 'golem.core.optimisers.common_optimizer.task.Task' imported but unused
Line 8:1: F401 'golem.core.optimisers.genetic.operators.base_mutations.MutationTypesEnum' imported but unused
Line 15:36: F821 undefined name 'CommonOptimizerParameters'
Line 22:45: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/nodes/old_crossover.py:

Line 7:36: F821 undefined name 'CommonOptimizerParameters'
Line 14:45: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/nodes/old_elitism.py:

Line 7:1: E303 too many blank lines (3)
Line 8:36: F821 undefined name 'CommonOptimizerParameters'
Line 14:45: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/nodes/old_inheritance.py:

Line 6:1: E302 expected 2 blank lines, found 1
Line 7:36: F821 undefined name 'CommonOptimizerParameters'
Line 13:45: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/nodes/old_mutation.py:

Line 7:36: F821 undefined name 'CommonOptimizerParameters'
Line 14:45: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/nodes/old_regularization.py:

Line 7:1: E303 too many blank lines (3)
Line 8:36: F821 undefined name 'CommonOptimizerParameters'
Line 15:45: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/nodes/old_selection.py:

Line 7:1: E303 too many blank lines (3)
Line 8:36: F821 undefined name 'CommonOptimizerParameters'
Line 13:45: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/nodes/rebuild_population.py:

Line 8:36: F821 undefined name 'CommonOptimizerParameters'
Line 14:45: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/nodes/uniqueness_check.py:

Line 1:1: F401 'copy.deepcopy' imported but unused
Line 2:1: F401 'random.choice' imported but unused
Line 4:1: F401 'typing.Optional' imported but unused
Line 4:1: F401 'typing.Dict' imported but unused
Line 4:1: F401 'typing.Callable' imported but unused
Line 7:1: F401 'golem.core.optimisers.common_optimizer.task.Task' imported but unused
Line 8:1: F401 'golem.core.optimisers.genetic.operators.base_mutations.base_mutations_repo' imported but unused
Line 8:1: F401 'golem.core.optimisers.genetic.operators.base_mutations.MutationTypesEnum' imported but unused
Line 9:1: F401 'golem.core.optimisers.graph.OptGraph' imported but unused
Line 10:1: F401 'golem.core.optimisers.opt_history_objects.individual.Individual' imported but unused
Line 11:1: F401 'golem.core.optimisers.opt_history_objects.parent_operator.ParentOperator' imported but unused
Line 15:36: F821 undefined name 'CommonOptimizerParameters'
Line 20:45: F821 undefined name 'CommonOptimizerParameters'
Line 47:1: W391 blank line at end of file

In the file golem/core/optimisers/common_optimizer/old_config.py:

Line 28:1: E302 expected 2 blank lines, found 0
Line 32:1: E305 expected 2 blank lines after class or function definition, found 1
Line 37:1: E302 expected 2 blank lines, found 1
Line 39:45: F821 undefined name 'CommonOptimizerParameters'
Line 43:1: E305 expected 2 blank lines after class or function definition, found 1
Line 54:1: E302 expected 2 blank lines, found 1
Line 61:1: E302 expected 2 blank lines, found 1
Line 66:1: E305 expected 2 blank lines after class or function definition, found 1
Line 74:45: F821 undefined name 'CommonOptimizerParameters'
Line 78:1: E305 expected 2 blank lines after class or function definition, found 1
Line 81:1: E302 expected 2 blank lines, found 1
Line 85:1: E305 expected 2 blank lines after class or function definition, found 1
Line 88:1: W391 blank line at end of file

In the file golem/core/optimisers/common_optimizer/runner.py:

Line 3:1: F401 'collections.deque' imported but unused
Line 4:1: F401 'copy.copy' imported but unused
Line 38:19: F821 undefined name 'Queue'
Line 39:22: F821 undefined name 'Queue'
Line 97:17: F841 local variable 'gen' is assigned to but never used

In the file golem/core/optimisers/common_optimizer/scheme.py:

Line 6:1: F401 'golem.utilities.utilities.determine_n_jobs' imported but unused

In the file golem/core/optimisers/common_optimizer/stage.py:

Line 33:46: F821 undefined name 'CommonOptimizerParameters'
Line 33:76: F821 undefined name 'CommonOptimizerParameters'
Line 35:31: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/task.py:

Line 4:1: F401 'typing.Any' imported but unused
Line 18:45: F821 undefined name 'CommonOptimizerParameters'
Line 26:1: E303 too many blank lines (3)
Line 39:45: F821 undefined name 'CommonOptimizerParameters'

In the file golem/core/optimisers/common_optimizer/temp/adaptive.py:

Line 1:1: F401 'typing.Any' imported but unused
Line 1:1: F401 'typing.Dict' imported but unused
Line 5:1: F401 'golem.core.optimisers.optimizer.OptimizationParameters' imported but unused
Line 5:1: F401 'golem.core.optimisers.optimizer.GraphGenerationParams' imported but unused
Line 5:1: F401 'golem.core.optimisers.optimizer.AlgorithmParameters' imported but unused
Line 11:1: F401 'golem.core.optimisers.genetic.operators.mutation.Mutation as OldMutation' imported but unused
Line 21:36: F821 undefined name 'CommonOptimizerParameters'
Line 30:45: F821 undefined name 'CommonOptimizerParameters'
Line 30:77: F821 undefined name 'CommonOptimizerParameters'

Comment last updated at 2023-12-14 12:40:45 UTC

codecov-commenter · 2023-12-08T09:09:48Z

Codecov Report

Attention: 387 lines in your changes are missing coverage. Please review.

Comparison is base (46b53a4) 71.94% compared to head (16a7a3e) 68.57%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
golem/core/optimisers/common_optimizer/runner.py	0.00%	78 Missing ⚠️
golem/core/optimisers/common_optimizer/config.py	0.00%	77 Missing ⚠️
...lem/core/optimisers/common_optimizer/old_config.py	0.00%	57 Missing ⚠️
...re/optimisers/common_optimizer/common_optimizer.py	0.00%	54 Missing ⚠️
golem/core/optimisers/common_optimizer/task.py	0.00%	35 Missing ⚠️
...misers/common_optimizer/common_optimizer_params.py	0.00%	32 Missing ⚠️
golem/core/optimisers/common_optimizer/scheme.py	0.00%	21 Missing ⚠️
golem/core/optimisers/common_optimizer/stage.py	0.00%	18 Missing ⚠️
golem/core/optimisers/common_optimizer/node.py	0.00%	15 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #247      +/-   ##
==========================================
- Coverage   71.94%   68.57%   -3.37%     
==========================================
  Files         136      145       +9     
  Lines        8127     8532     +405     
==========================================
+ Hits         5847     5851       +4     
- Misses       2280     2681     +401

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kasyanovse · 2024-10-13T08:47:37Z

ИМХО, этот PR на стадии MVP. Не хватает точных замеров прироста производительности в разных задачах, адаптированных под этот алгоритм мутаций (сейчас тут реализованы обертки) и нормальной системы кеширования.

А вообще, возможностей для дальнейшего развития тут так много, что проще с нуля написать.

kasyanovse added 3 commits December 6, 2023 17:57

wip

39b07fe

wip

2e2a318

wip

7a220f8

kasyanovse added research Experiments, hypothesis, research development adaptive Related to R&D of Adaptive features of the algorithm opt core Related to core logic of optimizer labels Dec 7, 2023

kasyanovse self-assigned this Dec 7, 2023

kasyanovse mentioned this pull request Dec 7, 2023

Parallel population processing #199

Closed

Lopa10ko self-assigned this Dec 7, 2023

kasyanovse and others added 2 commits December 7, 2023 18:20

add some operators

c5e7589

feat: add AdaptiveParametersTask Memento

0f7ef9d

Lopa10ko and others added 12 commits December 8, 2023 12:58

feat: add AdaptiveParameters

a449e1a

move temp directory

81d777e

fix: typo in AdaptiveParametersTask

2373f85

add some operators

4af20ee

feat: add some docstrings

afc608b

fix some errors

e219730

feat: AdaptivePopulationSize wrapper on Node

34f83e7

fix and add adaptive parameters to common_optimizer

e69fc2a

feat: add GraphDepth and OperatorsProb wrappers on Node

32dcdb4

fix scheme for existing optimiser

4ca1d5d

fix probs

94cc8e7

uniqueness node and some fixes

16a7a3e

maypink requested review from YamLyubov and maypink December 19, 2023 13:07

maypink mentioned this pull request Dec 19, 2023

Release 0.4.1 #254

Open

8 tasks

nicl-nno mentioned this pull request Oct 14, 2024

enh: Better parallelization of fitness evaluations aimclub/FEDOT#1340

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallel optimizer #247

Parallel optimizer #247

kasyanovse commented Dec 7, 2023 •

edited

Loading

pep8speaks commented Dec 7, 2023 •

edited

Loading

codecov-commenter commented Dec 8, 2023 •

edited

Loading

kasyanovse commented Oct 13, 2024

Parallel optimizer #247

Are you sure you want to change the base?

Parallel optimizer #247

Conversation

kasyanovse commented Dec 7, 2023 • edited Loading

Этап 1. Воссоздание существующего генетического оптимизатора с использованием нового подхода

Этап 2. Добавление бандитов для мутаций

pep8speaks commented Dec 7, 2023 • edited Loading

Comment last updated at 2023-12-14 12:40:45 UTC

codecov-commenter commented Dec 8, 2023 • edited Loading

Codecov Report

kasyanovse commented Oct 13, 2024

kasyanovse commented Dec 7, 2023 •

edited

Loading

pep8speaks commented Dec 7, 2023 •

edited

Loading

codecov-commenter commented Dec 8, 2023 •

edited

Loading