ENH: SPMD interface for IncrementalPCA

[olegkkruglov]

Description

• Added SPMD interface for IncrementalEmpiricalCovariance
• Changed policy saving workflow, now queue is saved to attributes instead of policy. It is necessary because finalize_fit requires spmd_policy, but partial_fit requires data_parallel_policy on oneDAL side
• finalize_fit now uses provided queue for computations on onedal4py side.
• Contains some content from TEST: test coverage for sklearnex SPMD ifaces #1777 for test implementation

---

Checklist to comply with before moving PR from draft:

PR completeness and readability

• I have reviewed my changes thoroughly before submitting this pull request.
• I have commented my code, particularly in hard-to-understand areas.
• I have updated the documentation to reflect the changes or created a separate PR with update and provided its number in the description, if necessary.
• Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
• I have added a respective label(s) to PR if I have a permission for that.
• I have resolved any merge conflicts that might occur with the base branch.

Testing

• The unit tests pass successfully.
• I have run it locally and tested the changes extensively.

Performance

• I have measured performance for affected algorithms using scikit-learn_bench and provided at least summary table with measured data, if performance change is expected.
• I have provided justification why performance has changed or why changes are not expected.

[samir-nasibli]

@olegkkruglov please rebase your branch

[ethanglaser]

/intelci: run

[ethanglaser]

https://intel-ci.intel.com/ef5f226d-b65f-f196-993d-a4bf010d0e2e

[olegkkruglov]

/intelci: run

[samir-nasibli]

Description

• Added SPMD interface for IncrementalEmpiricalCovariance
• Changed policy saving workflow, now queue is saved to attributes instead of policy. It is necessary because finalize_fit requires spmd_policy, but partial_fit requires data_parallel_policy on oneDAL side
• finalize_fit now uses provided queue for computations on onedal4py side.
• Contains some content from TEST: test coverage for sklearnex SPMD ifaces #1777 for test implementation

Checklist to comply with before moving PR from draft:

PR completeness and readability

• I have reviewed my changes thoroughly before submitting this pull request.
• I have commented my code, particularly in hard-to-understand areas.
• I have updated the documentation to reflect the changes or created a separate PR with update and provided its number in the description, if necessary.
• Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
• I have added a respective label(s) to PR if I have a permission for that.
• I have resolved any merge conflicts that might occur with the base branch.

Testing

• The unit tests pass successfully.
• I have run it locally and tested the changes extensively.

Performance

• I have measured performance for affected algorithms using scikit-learn_bench and provided at least summary table with measured data, if performance change is expected.
• I have provided justification why performance has changed or why changes are not expected.

@olegkkruglov Is it possible to update and use global queue in scikit-learn-intelex for the finalize_fit? Could be done as a followup for all Incr algos, just to remove storing queue in models itself.

[samir-nasibli]

@olegkkruglov All CI checks should be included to the PR before making it ready for the review. For SPMD algos this is mandatory to have internal CI run.

[samir-nasibli]

/intelci: run

[olegkkruglov]

/intelci: run

[olegkkruglov]

https://intel-ci.intel.com/ef66d2e7-6006-f173-ab00-a4bf010d0e2e

[olegkkruglov]

@olegkkruglov Is it possible to update and use global queue in scikit-learn-intelex for the finalize_fit? Could be done as a followup for all Incr algos, just to remove storing queue in models itself

Probably it is possible but what's the point? Serialization problem is already resolved and I don't see any other problems with it

[icfaust]

I can't comment much on the implementation of the spmd object side of things, since I really didn't review the initial IncrementalPCA PR. My focus was on similar issue as the other Incremental SPMD PRs for 2025/ testing. I assume the _create_model method comes from that implementation, and that it was justified for sklearn conformance reasons.

[samir-nasibli]

Please rebase your branch and run CI

[icfaust]

@olegkkruglov ping me when you want another review.

[samir-nasibli]

Thank you! Expecting other reviewers approvals.
Assuming green CI

[olegkkruglov]

/intelci: run

[icfaust]

Merge whenever you are ready.

[ethanglaser]

Interesting that spmd incremental pca tests are passing but not normal pca spmd