[ci] fix flaky dask tests on GPU

[jameslamb]

Opening this to try to narrow down the flakiness in the GPU tests.

Reference #5677 (comment).

[jameslamb]

4 of the gpu builds failed in the first commit I pushed here, all on the same test.

../tests/python_package_test/test_engine.py::test_multiclass_custom_eval[False] PASSED [ 73%]
../tests/python_package_test/test_engine.py::test_model_size /__w/1/s/.ci/test.sh: line 251:  7934 Killed                  pytest -vvv $BUILD_DIRECTORY/tests

Interestingly, all of those jobs made it through all of the Dask tests.

[jameslamb]

The success rate went up after skipping that one test_engine.py::test_model_size, but now I see another Dask test failure (build link

FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-goss-group1-array]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-goss-group1-dataframe]

=================================== FAILURES ===================================
_____________________ test_ranker[data-goss-group1-array] ______________________

output = 'array', group = [5, 5, 5, 10, 10, 10, ...], boosting_type = 'goss'
tree_learner = 'data'
cluster = LocalCluster(f1e3d9c7, 'tcp://127.0.0.1:36971', workers=1, threads=2, memory=3.89 GiB)

    @pytest.mark.parametrize('output', ['array', 'dataframe', 'dataframe-with-categorical'])
    @pytest.mark.parametrize('group', [None, group_sizes])
    @pytest.mark.parametrize('boosting_type', boosting_types)
    @pytest.mark.parametrize('tree_learner', distributed_training_algorithms)
    def test_ranker(output, group, boosting_type, tree_learner, cluster):
        with Client(cluster) as client:
            if output == 'dataframe-with-categorical':
                X, y, w, g, dX, dy, dw, dg = _create_data(
                    objective='ranking',
                    output=output,
                    group=group,
                    n_features=1,
                    n_informative=1
                )
            else:
                X, y, w, g, dX, dy, dw, dg = _create_data(
                    objective='ranking',
                    output=output,
                    group=group
               )


_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
E   lightgbm.basic.LightGBMError: Socket recv error, Connection reset by peer (code: 104)

/home/AzDevOps_azpcontainer/.local/lib/python3.7/site-packages/lightgbm/basic.py:177: LightGBMError

[jameslamb]

Ok, just skipping the Dask learning-to-rank + GOSS test didn't help. All of the other Dask learning-to-rank GPU tests failed too. (build link)

=========================== short test summary info ============================
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-gbdt-None-array]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-gbdt-None-dataframe]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-gbdt-None-dataframe-with-categorical]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-gbdt-group1-array]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-gbdt-group1-dataframe]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-gbdt-group1-dataframe-with-categorical]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-dart-None-array]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-dart-None-dataframe]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-dart-None-dataframe-with-categorical]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-dart-group1-array]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-dart-group1-dataframe]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-dart-group1-dataframe-with-categorical]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-goss-None-array]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-goss-None-dataframe]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-goss-None-dataframe-with-categorical]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-goss-group1-array]
FAILED ../tests/python_package_test/test_dask.py::test_ranker[data-goss-group1-dataframe]

[jameslamb]

This has been open for a long time without new activity, and there are several other higher-priority issues I'm spending my limited time on right now. I'm closing this to focus on other things.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.