SyntaxError: Unexpected end of JSON input at JSON.parse (<anonymous>) at NamespaceFS.complete_object_upload #8620
Comments
|
Hi, Summary so far:
Additional Information:
I will share something that I saw: But in the bucket files there is no
But when looking at the file:
From what I saw the structure name of the files is
I saw only 1 file (although I would expect to see a directory for each operation of a complete multipart upload that failed).
{
"bucket": "newbucket-warp-ha-efix-downstream-19dec2024-defrecreate",
"key": "4fTpXE7s/208.ezMZmJlL1AQyxVNY.rnd",
"content_type": "application/octet-stream",
"xattr": {},
"storage_class": "STANDARD",
"obj_id": "2a8e302c-fa5f-4a49-88dd-c70399590c95",
"mpu_path": "/ibm/fvt_fs/s3user-18002-dir/newbucket-warp-ha-efix-downstream-19dec2024-defrecreate/.noobaa-nsfs_6764165e0bf4eb72dad51b45/multipart-uploads/2a8e302c-fa5f-4a49-88dd-c70399590c95",
"source_stream": null
}In this example, we can see a valid JSON that can be parsed. noobaa-core/src/sdk/namespace_fs.js Lines 1649 to 1672 in 7c70730 To make sure that I found all the existing files that have
looking at this upload ID we didn't see it is related to the mentioned error: We also tried to look at the second node with The rationale for this search is to understand if the directory of multipart upload of the upload ID was deleted as part of this operation: noobaa-core/src/sdk/namespace_fs.js Lines 1911 to 1916 in 7bc814b The multipart upload directory is deleted either from the abort (as you can see above) or from the complete, but we didn't find those keys in either of these operations. noobaa-core/src/sdk/namespace_fs.js Line 1884 in 7bc814b
|
|
Hi @rkomandu, What I can suggest is when trying to reproduce it to run a shell script that would display the json files that we have once in a while with a timestamp (and save this output in logs). To check the idea on my local machine I ran something like that on a script that was created in the directory of the bucket Partial output for example: We can see in the output that this is a valid JSON. |
|
Hi @rkomandu, I also added log printing lines in the functions so they will be printed (as a string without parsing it to JSON). For example: I removed the first curly bracket ( And it is printed before the action fails: |
|
Hi @rkomandu, |
Environment info
Noobaa d/s rpm noobaa-core-5.17.2-20241217.el9.ppc64le
Actual behavior
When running Warp IO workload and performing the HA functionality with assert, sigsegv to mmfs daemon, the noobaa log observed the following messages
Warp run
./warp mixed --noclear --insecure --duration 120m --host .com:6443 --access-key 5gzVKZdN9XerVoPtImPr --secret-key evxl0frzdnk5rTYkj0fg0WkDxAcwTaYdgEVAoH0A --tls --obj.size 256M --bucket newbucket-warp-ha-efix-downstream-19dec2024-defrecreate
noobaa.log
Warp run started at 08:04 -> 10:04 for 2hrs run
ls -lrt warp-mixed-2024-12-19[080323]-1MuL.csv.zst
-rw-r--r--. 1 root root 1143577 Dec 19 10:04 'warp-mixed-2024-12-19[080323]-1MuL.csv.zst'
Expected behavior
Error w/r/t JSON output is concerning, does it has a problem w/r/t metadata access of JSON output ?
Steps to reproduce
Warp run and perform HA of GPFS assert , sigsegv in a loop
More information - Screenshots / Logs / Other output
Output logs uploaded to box folder https://ibm.ent.box.com/folder/299558120602
The text was updated successfully, but these errors were encountered: