Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to checksum speed and uploading large files #2627

Open
scott-hpe opened this issue Jul 29, 2024 · 0 comments
Open

Improvements to checksum speed and uploading large files #2627

scott-hpe opened this issue Jul 29, 2024 · 0 comments
Labels
feature request New feature or request

Comments

@scott-hpe
Copy link

Is your feature request related to a problem? Please describe.
When uploading using jf rt upload, the checksum of an artifact is calculated before sending the data. If the artifact is very large, the checksum takes a long time to calculate. After a long period of checksum calculation, the upload will fail becuase the size of the artifact exceeds the allowed max file size on Artifactory SaaS.

Describe the solution you'd like to see
Ideally the following would happen:

  1. Validate that an artifact of the requested size can be uploaded before doing local checksum calculations
  2. A faster method of local checksums. (eg, larger read buffer size when running the checksum)

Describe alternatives you've considered
Looked for an option to skip checksums, but that's not ideal as checksums provide integrety value.

Additional context

In this case the local artifact that exceded the configured Artifactory SaaS max file size was 182GB. Checksum calculations took several minutes before finally failing.

esroot@op360-g10s14-vm01:~$ cat /home/esroot/.jfrog/logs/jfrog-cli.2024-07-29.05-41-46.2313818.log
[Debug] Usage Report: Sending info...
[Debug] Sending HTTP GET request to: https://hcss.jfrog.io/artifactory/api/system/version
[Info] [Thread 9] Uploading: 0.9/0.9.0/alpha-rc3/signatures/sc-onprem-pce-0.9.0-alpha.56+77.nightly-build.tar.sig
[Info] [Thread 2] Uploading: 0.9/0.9.0/alpha-rc3/signatures/glcp-onprem-combined-0.9.0-alpha.291+330-fips.ova.sig
[Info] [Thread 4] Uploading: 0.9/0.9.0/alpha-rc3/signatures/pceall-checksum.txt.sig
[Info] [Thread 6] Uploading: 0.9/0.9.0/alpha-rc3/sc-onprem-pceall-0.9.0-alpha.56+77.nightly-build.tar
[Info] [Thread 3] Uploading: 0.9/0.9.0/alpha-rc3/signatures/HewlettPackardEnterpriseCompany2023-3-4.public
[Info] [Thread 7] Uploading: 0.9/0.9.0/alpha-rc3/pceall-manifest.yaml
[Info] [Thread 5] Uploading: 0.9/0.9.0/alpha-rc3/sc-onprem-pce-0.9.0-alpha.56+77.nightly-build.tar
[Info] [Thread 0] Uploading: 0.9/0.9.0/alpha-rc3/pceall-checker.yaml
[Info] [Thread 1] Uploading: 0.9/0.9.0/alpha-rc3/pceall-checksum.txt
[Info] [Thread 8] Uploading: 0.9/0.9.0/alpha-rc3/pce-manifest.yaml
[Debug] Artifactory response: 200 
[Debug] JFrog Artifactory version is: 7.90.5
[Debug] Sending HTTP POST request to: https://hcss.jfrog.io/artifactory/api/system/usage
[Debug] [Thread 7] Artifactory response: 201  
[Info] [Thread 7] Uploading: 0.9/0.9.0/alpha-rc3/pce-checksum.txt
[Debug] [Thread 4] Artifactory response: 201  
[Info] [Thread 4] Uploading: 0.9/0.9.0/alpha-rc3/pce-checker.yaml
[Debug] [Thread 8] Artifactory response: 201  
[Info] [Thread 8] Uploading: 0.9/0.9.0/alpha-rc3/glcp-onprem-combined-0.9.0-alpha.291+330-fips.ova
[Debug] [Thread 3] Artifactory response: 201  
[Info] [Thread 3] Uploading: 0.9/0.9.0/alpha-rc3/CHANGELOG-0.9.0-alpha.56+77.nightly-build.pce.html
[Debug] [Thread 2] Artifactory response: 201  
[Debug] Sending HTTP PUT request to: https://hcss.jfrog.io/artifactory/zion-privatecloud/SC/0.9/0.9.0/alpha-rc3/CHANGELOG-0.9.0-alpha.56+77.nightly-build.pce.html
[Debug] [Thread 9] Artifactory response: 201  
[Debug] [Thread 0] Artifactory response: 201  
[Debug] [Thread 1] Artifactory response: 201  
[Debug] [Thread 3] Artifactory response: 201   (Checksum deploy)
[Debug] [Thread 4] Artifactory response: 201  
[Debug] [Thread 7] Artifactory response: 201  
[Debug] Sending HTTP PUT request to: https://hcss.jfrog.io/artifactory/zion-privatecloud/SC/0.9/0.9.0/alpha-rc3/glcp-onprem-combined-0.9.0-alpha.291+330-fips.ova
[Debug] [Thread 8] Artifactory response: 201   (Checksum deploy)
[Debug] Sending HTTP PUT request to: https://hcss.jfrog.io/artifactory/zion-privatecloud/SC/0.9/0.9.0/alpha-rc3/sc-onprem-pce-0.9.0-alpha.56+77.nightly-build.tar
[Debug] [Thread 5] Artifactory response: 201   (Checksum deploy)
[Debug] Sending HTTP PUT request to: https://hcss.jfrog.io/artifactory/zion-privatecloud/SC/0.9/0.9.0/alpha-rc3/sc-onprem-pceall-0.9.0-alpha.56+77.nightly-build.tar
[Debug] Sending HTTP GET request to: https://hcss.jfrog.io/artifactory/api/system/version
[Debug] Artifactory response: 200 
[Debug] JFrog Artifactory version is: 7.90.5
[Debug] Sending HTTP GET request to: https://hcss.jfrog.io/artifactory/api/v1/uploads/config
[Debug] Artifactory response: {
  "supported" : false
} 200 
[Warn] [Thread 6] (Attempt 1) - Failure occurred while uploading to https://hcss.jfrog.io/artifactory/zion-privatecloud/SC/0.9/0.9.0/alpha-rc3/sc-onprem-pceall-0.9.0-alpha.56+77.nightly-build.tar: server response: 413 Request Entity Too Large
<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>

[Warn] [Thread 6] (Attempt 2) - Failure occurred while uploading to https://hcss.jfrog.io/artifactory/zion-privatecloud/SC/0.9/0.9.0/alpha-rc3/sc-onprem-pceall-0.9.0-alpha.56+77.nightly-build.tar: server response: 413 Request Entity Too Large
<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>

[Warn] [Thread 6] (Attempt 3) - Failure occurred while uploading to https://hcss.jfrog.io/artifactory/zion-privatecloud/SC/0.9/0.9.0/alpha-rc3/sc-onprem-pceall-0.9.0-alpha.56+77.nightly-build.tar: server response: 413 Request Entity Too Large
<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>

[Warn] [Thread 6] (Attempt 4) - Failure occurred while uploading to https://hcss.jfrog.io/artifactory/zion-privatecloud/SC/0.9/0.9.0/alpha-rc3/sc-onprem-pceall-0.9.0-alpha.56+77.nightly-build.tar: server response: 413 Request Entity Too Large
<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>

[Info] [Thread 6]  executor timeout after 3 attempts with 0 milliseconds wait intervals
[Error] server response: 413 Request Entity Too Large
<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>

[Debug] Uploaded 13 artifacts.
[Error] Failed uploading 1 artifacts.
[Error] server response: 413 Request Entity Too Large
<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>


@scott-hpe scott-hpe added the feature request New feature or request label Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant