-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ttnn.full and ttnn.ones operations fail by assert valid_page_size on some dimensions #8633
Comments
Hi @nemanjagrujic , I have ran the test the issue is reproducible. I have a question here. @eyonland any comment here |
Hello @ruthreshx, most of ttnn ops work correctly with tile layout even if shape is not tile (almost all ops). There is some automatic handling of that case. |
Hi @nemanjagrujic , So that I'm able to track the function calls over there to compare such issues between those ops. |
Not sure what do you mean by Tile_one. But you can checkout branch You can try:
|
Hi @nemanjagrujic , @eyonland , Yes, Reg the abs, it should pass, but Full Op has a restriction from the kernel side.
Please find the full op implementation, where it has an TT assert link |
@ruthreshx, @eyonland Well, that seems like an opportunity for improvement then. If only few ops have that restriction it can (unnecessarly) complicate user workflow. For instance even |
Hi @nemanjagrujic , @eyonland , Either we need to change the shape to Tile or we need to redirect this to the respective author who implemented this op to add such support. Thank you! |
@ruthreshx, @eyonland, well I recommend redirecting.. |
Hi @nemanjagrujic , @eyonland , Case 2: I have raised the PR for TT_Fatal to avoid the issue been capturing in debug mode. |
Merged PR #12921 to main. can we close this issue @nemanjagrujic |
@VirdhatchaniKN Have in mind that the limitation is not added to docs: https://docs.tenstorrent.com/tt-metal/latest/ttnn/ttnn/api/ttnn.full.html and, example given in docs does not work since we have this limitation. |
Hi @nemanjagrujic , |
Hi @nemanjagrujic , I've updated the docs like shown below in #13808 |
Merged #13808 |
Closing Issue |
When testing ttnn ops with TILE layout and random shapes like [4, 7, 21, 133] most of the operations work correctly.
But ttnn.full and ttnn.ones operations fail with assertion like:
which is followed by crash:
or sometimes:
Note that similar operation ttnn.zeros works in same conditions!
Problem is observed on both GS and WH cards.
To Reproduce
Steps to reproduce the behavior:
ngrujic/op_bug_unit_tests
(soon to be merged intomain
).test_eltwise_full_and_ones.py
using this command:Expected behavior
There are few test cases presented in the unit test, which are failing with:
which is followed by crash:
Note to observe this crash (and not only assertion fail), you must run sweep:
Running sweeps
To get additional information and results for different combinations of input shapes, types, layouts and memory configs for which this operation was tested you can also run locally sweeps and check the results. To do this you should:
pytest tests/ttnn/python_api_testing/sweep_tests/run_sweep_test.py --input-path tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/wormhole/ttnn_eltwise_full_test.yaml --input-method cli --cli-input results_ttnn_full
There are more sweeps which you can try by changing the above command to target files:
The text was updated successfully, but these errors were encountered: