Failure cloning git repos in testing

[pshipton]

osx1014-x86-1
https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.system_x86-64_mac_Nightly_testList_0/253

03:51:48  git clone --depth 1 -b master https://github.com/eclipse-openj9/openj9.git
03:52:04  update to openj9 sha: 194ae711901aeec136e9679c660c2b6a9be086cb
03:52:04  git fetch -q --unshallow
03:52:39  error: RPC failed; curl 18 transfer closed with outstanding read data remaining

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_0/236

04:35:07  git clone --depth 1 -b master https://github.com/eclipse-openj9/openj9.git
04:35:45  update to openj9 sha: 194ae711901aeec136e9679c660c2b6a9be086cb
04:35:45  git fetch -q --unshallow
04:36:11  error: RPC failed; curl 18 transfer closed with outstanding read data remaining

mac11-aarch64-2
https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_aarch64_mac_Nightly_testList_1/30

01:40:40  git clone --depth 1 -b master https://github.com/eclipse-openj9/openj9.git
01:40:49  error: 1279 bytes of body are still expected

win2012-x86-6
https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_sanity.functional_x86-64_windows_Nightly_testList_1/43

Fetching upstream changes from https://github.com/adoptium/aqa-tests.git
 > git --version # timeout=10
 > git --version # 'git version 2.17.0'
 > git fetch --tags --progress -- https://github.com/adoptium/aqa-tests.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/adoptium/aqa-tests.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: object directory /cygdrive/f/Users/jenkins/workspace/Test_openjdk18_j9_sanity.functional_x86-64_windows_Nightly_testList_1/aqa-tests/.git/objects/C:/cygwin64/home/jenkins/openjdk_cache/objects does not exist; check .git/objects/info/alternates.

cent7-ppcle-5
https://openj9-jenkins.osuosl.org/job/Test_openjdknext_j9_extended.system_ppc64le_linux_Nightly_testList_1/30

04:16:17  git clone --depth 1 -b master https://github.com/eclipse-openj9/openj9.git
04:16:47  update to openj9 sha: 194ae711901aeec136e9679c660c2b6a9be086cb
04:16:47  git fetch -q --unshallow
04:17:12  error: RPC failed; curl 18 transfer closed with outstanding read data remaining

[pshipton]

@llxia fyi. We have problems cloning git repos off and on, but it's been bad the last two nights. I've recorded the failures from last night. There were probably more the previous night. Maybe we could do more to retry to avoid these failures?

[llxia]

This is a network issue. There are several suggestions online [1]:

1. improve network/connection :)
2. --depth 1 and --unshallow (we are doing this already)
3. switch to SSH
4. increment buffer size

I am not sure there is much we can do with improving the network. Should we try to switch to SSH?

[1]: https://stackoverflow.com/questions/38618885/error-rpc-failed-curl-transfer-closed-with-outstanding-read-data-remaining

[pshipton]

@AdamBrousseau fyi in case you have any info.

Should we try to switch to SSH?

Sure? It can't hurt to try, except the time invested in making the switch. In the end what may work best is to wait for a few minutes and then try again.

[pshipton]

I just realized that internally we use ssh but it also fails regularly lately. Moving to SSH may be a waste of time.

For reference, see the Trigger-Product-Builds jenkins job.

[pshipton]

Although it may be a different problem.

[llxia]

yes, internal git seems very slow lately. I feel the internal problem is on the server-side as UI response is very slow as well.

@renfeiw could you look into adding the retry? It should be very similar to https://github.com/adoptium/aqa-tests/blob/master/get.sh#L265

[pshipton]

Haven't seen this for a while, I expect it's resolved.

[llxia]

Just for the record, retry has added in for different test repos.
adoptium/aqa-tests#3825
adoptium/aqa-tests#3774
adoptium/aqa-tests#3738
adoptium/aqa-tests#3633
adoptium/aqa-tests#3609