Replies: 5 comments 24 replies
-
|
@myc724 Sorry for the regression. There are hundreds of changes between these two, can you try other releases between then to narrow down the range? |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
lsof found the port but curl showed "No match" as follow. Any other command or option that I can try? Thanks! lsof -p 111611 | grep LISTEN curl localhost:9567/debug/pprof/goroutine?debug=1 |
Beta Was this translation helpful? Give feedback.
-
|
Just realized you're looking for stack trace of pprof and gops but the mount was done using "--no-agent". |
Beta Was this translation helpful? Give feedback.
-
|
The run with pprof and gops agents took much longer but still reached the same stage (after ~3.5 hours) where +1000 machines reading a huge library (excessive amount of small files) together and then small writes (for bookkeeping) from each compile process. But, I did not get anything from curl. Please let me know what else I can try. Thanks!
|
Beta Was this translation helpful? Give feedback.
-
|
That's weird, both 6061 and 9567 should work. What's your version curl? Mine is Can you try |
Beta Was this translation helpful? Give feedback.
-
I will see if our IT can update to the matching curl to the latest one. But, it might take a while since each of our machine follows certain predefined spec. Does this version make a lot of difference? Let me know if there's some way that I can bypass this hrudle. Thanks! BTW, I locally compiled the most recent curl (from git) but it still returns no match.
|
Beta Was this translation helpful? Give feedback.
-
|
I found the issue with curl. I need to quotes and feed curl. curl "localhost:6061/debug/pprof/goroutine?debug=1" |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
For another port ... curl "localhost:9567/debug/pprof/goroutine?debug=1" |
Beta Was this translation helpful? Give feedback.
-
|
These two port are serving the same internal status. As the list of goroutine shows, there was no deadlock in JuiceFS at that time. Can you reproduce the behavior and get the list of goroutines when the compiler hang? Here is the way to find out where the compile hang:
|
Beta Was this translation helpful? Give feedback.
-
|
I search all these +1000 machines' juicefs log and didn't find any ERROR or FATAL. Also, can you elaborate what this change trying to achieve? Thanks! |
Beta Was this translation helpful? Give feedback.
10 replies
-
|
Is there anything in the logging on the workers that hanged? BTW, before #681, TiKV is already supported, can you try TiKV with the version before #681? Was the Redis overloaded when you ran the compile workload? |
Beta Was this translation helpful? Give feedback.
-
|
I didn't use TiKV in these debugging exercises. I didn't find any "FATAL|ERROR|not enough bytes|interrupted|slow operation" from the JuiceFS log of all the nodes that workers that hanged. FYI. I posted the JuiceFS + TiKV experiment in a different discussion. |
Beta Was this translation helpful? Give feedback.
-
|
curl on 2 ports and grep vfs. Let me know if full log is needed. |
Beta Was this translation helpful? Give feedback.
-
|
These looks good, there are background thread. |
Beta Was this translation helpful? Give feedback.
-
|
The callgraph is [app]open(xxx) -> [kernel]fuse_wait_answer -> [juicefs]fuse.process -> [juicefs] vfs.Open() -> [juicefs] meta.Open() OR [app]read(xxx) -> [kernel]wait_on_mutex -> [kernel, another thread]fuse_wait_answer -> [juicefs]fuse.process -> [juicefs] vfs.Read() -> [juicefs] vfs.(*datareader).Read() -> XXX So, when there is a syscall hang on JuiceFS, we will see a goroutine having a line in vfs.XXX, the |
Beta Was this translation helpful? Give feedback.
-
|
These're from a worker machine which did not respond for a very long hours. The lines of trace seems different. curl "localhost:6061/debug/pprof/goroutine?debug=1" |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
no deadlock here also. |
Beta Was this translation helpful? Give feedback.
-
|
curl "localhost:9567/debug/pprof/goroutine?debug=1" |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
For a deadlock, you will see at least one goroutine wait in vfs.XXX() (for example, |
Beta Was this translation helpful? Give feedback.
-
|
Did you see any warning about "not enough bytes" or "interrupted" or "slow operation" in the logging file? |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We want to switch JuiceFS 0.16 to deploy TiKV. However, we have a test which deployed 1250 machines to read and compile 140GB of source code (1.2 million files) then write 73GB object files (114 thousand files) in parallel. After switching JuiceFS from 0.11.0-37 to 0.16 with identical format/mount configuration (eg. --block-size, --cache-size, etc.), the overall runtime dropped by a lot. Our study indicated that many JuiceFS client operations timeout (ie. read/write failures) forced our compiler processes to find different hosts to recompile the failure components. We’ve tried various knobs (eg, --get-timeout, --put-timeout, --io-retries, etc. ) but none of them seems to help.
We noticed that, for the format command, JuiceFS 0.11.0-37 default “--compress lz4” while JuiceFS 0.16 default “--compress none”. After applying lz4 compression, we found that, if we format a volume using the JuiceFS 0.16 but mount it to all machines using JuiceFS 0.11.0-37, this test can match the performance of the earlier and fastest run which fully deployed JuiceFS 0.11.0-37. We studied the mount command source code but didn’t find any obvious default setup change. Is there any default configuration being adjusted for the mount command from JuiceFS 0.11.0-37 to JuiceFS 0.16? Just want to see whether we can try to revert it to match for this test. Thanks!
the original run
juicefs version 0.11.0-37 ( 6127863)
the new run
juicefs version 0.16-dev (2021-08-02 5d93e58)
both runs using the same Redis for metadata as well as object-storage
Redis server v=6.2.2 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=3e62cb2e1fe34d7f
Beta Was this translation helpful? Give feedback.
All reactions