Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug Report] ttnn.full input shape causes Segfault with TILE_LAYOUT #15030

Open
kevinwuTT opened this issue Nov 14, 2024 · 2 comments
Open

[Bug Report] ttnn.full input shape causes Segfault with TILE_LAYOUT #15030

kevinwuTT opened this issue Nov 14, 2024 · 2 comments
Labels
bug Something isn't working P0 pytorch-compiler

Comments

@kevinwuTT
Copy link

Describe the bug

Calling ttnn.full with this shape (1, 50257) with TILE_LAYOUT on device gives a segfault. (This shape is used in GPTNeo.)

To Reproduce

import ttnn

with ttnn.manage_device(device_id=0) as device:
    ttnn_full = ttnn.full((1, 50257), layout = ttnn.TILE_LAYOUT, fill_value = 1.0, device = device)
Segmentation fault (core dumped)

However, if I have ttnn.full default to ROW_MAJOR_LAYOUT and call ttnn.to_layout to TILE_LAYOUT after, I do not get any errors.

import ttnn

with ttnn.manage_device(device_id=0) as device:
    ttnn_full = ttnn.full((1, 50257), fill_value = 1.0, device = device)
    to_layout = ttnn.to_layout(ttnn_full, layout = ttnn.TILE_LAYOUT)

Expected behavior
ttnn.full alone should support this shape (1, 50257) with TILE_LAYOUT since splitting into two ops work.

Please complete the following environment information:

  • OS: Ubuntu 20.04 Grayskull
  • Version of software (eg. commit): tt-metal: tags/v0.53.0-rc39

With a previous version, v0.53.0-rc35 there is no segfault, but this error:

RuntimeError: TT_FATAL @ /tmp/build-via-sdist-x_q5c5zg/metal_libs-0.53.0rc35+grayskull/ttnn/cpp/ttnn/operations/numpy/functions.hpp:66: shape[-1] % tt::constants::TILE_WIDTH == 0
info:
TILE layout requires width dimension to be multiple of 32

Might be related to this: #14871

@ayerofieiev-tt
Copy link
Member

Setting to p0. We need help to get this resolved asap. We picked up latest metal wheel and ci got broken. Thank you!

@KalaivaniMCW
Copy link
Contributor

  1. ttnn.full uses host operation and required shape is given directly, so padding is not handled.

  2. For TILE layout, HW should be multiples of 32 as mentioned in the doc
    Image

  3. The TT_FATAL was removed recently as part of #0: Add support for 0-volume and 1-volume tensors for ttnn::add #14611 hence the segmentation fault instead of runtime error
    Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P0 pytorch-compiler
Projects
Status: No status
Development

No branches or pull requests

3 participants