Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rpi 5 NVMe boots only if USB-MSD is there #629

Open
doubleuerner opened this issue Nov 7, 2024 · 13 comments
Open

Rpi 5 NVMe boots only if USB-MSD is there #629

doubleuerner opened this issue Nov 7, 2024 · 13 comments
Assignees

Comments

@doubleuerner
Copy link

doubleuerner commented Nov 7, 2024

Describe the bug

I am struggling with booting from an NVMe disk (Toshiba KBG40ZNS256G) and after a couple of mistakes and spending fun but also confusing hours i think i discoverd a bug or a symptom caused by doing something wrong or missing something , or it is a bug for real. Who knows? I'll try to explain:

I found out that simply using dd an image to a NVMen0 (4096 sector size) with bs=515(default) gives a weird situation. But the SD card copier tool helped to overcome.

Some info about my setup:

Raspberry Pi 5 Model B Rev 1.0
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"

picture:

I've cloned USB-MSD into NVMe disk using the "SD card Copier" tool.
I changed the boot order,

BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf461
NET_INSTALL_AT_POWER_ON=1

During power up SD card is not present, USB thumbdrive neither.

I would expect to boot from NVMe, since i changed the boot order:
.

What i get is a pi that doesnt boot and in the diagnotic screen:

picture:

nvme: error 10

But now the suprise:

But powering up with a USB-MSD in one of thew usb ports..., it boots from NVMe, it's so fast you could even not see any 'bios' screen or diagnostic screen.

I am happy my goal has been reached, but only to trick the pi with a USB-MSD first. after start up it is not mounted and i can just remove it.

root@rpi5-usb:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 58.6G 0 disk
├─sda1 8:1 1 512M 0 part
└─sda2 8:2 1 58.1G 0 part
nvme0n1 259:0 0 238.5G 0 disk
├─nvme0n1p1 259:1 0 4G 0 part /boot/firmware
└─nvme0n1p2 259:2 0 234.4G 0 part /

It's not the case a config.txt or cmdline.txt is read from USB-MSD, because it requires then to do something with UUID or correct devices (eg sda / sdb)

The last thing i want now is to get rid of the USB-MSD to be injected as a prerequisit to boot the NVMe ... or its a bug may be?

Steps to reproduce the behaviour

NVMe disk partitioned and having OS installed.

power up without SD ; but with USB-MSD : it boot from NVMe disk

power up without SD ; and NO USB-MSD : nvme error 10, boot fails and others (SD, USB ) are try out.
Afbeelding van WhatsApp op 2024-11-07 om 14 43 36_ce56ab00
Afbeelding van WhatsApp op 2024-11-07 om 14 24 13_01ec40ac

Device (s)

Raspberry Pi 5

Bootloader configuration.

root@rpi5-usb:/var/lib/raspberrypi/bootloader/backup# cat pieeprom-backup-20241107-140225.conf
[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf461
NET_INSTALL_AT_POWER_ON=1

root@rpi5-usb:/var/lib/raspberrypi/bootloader/backup# rpi-eeprom-config
[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf461
NET_INSTALL_AT_POWER_ON=1

root@rpi5-usb:/var/lib/raspberrypi/bootloader/backup# ls -alh
total 20K
drwxr-xr-x 2 root root 4.0K Nov 7 14:02 .
drwxr-xr-x 3 root root 4.0K Oct 22 15:05 ..
-rw------- 1 root root 56 Nov 7 13:59 pieeprom-backup-20241107-135912.conf
-rw------- 1 root root 82 Nov 7 13:59 pieeprom-backup-20241107-135940.conf
-rw------- 1 root root 82 Nov 7 14:02 pieeprom-backup-20241107-140225.conf
root@rpi5-usb:/var/lib/raspberrypi/bootloader/backup#

root@rpi5-usb:~# rpi-eeprom-update
BOOTLOADER: up to date
CURRENT: Tue 5 Nov 12:38:12 UTC 2024 (1730810292)
LATEST: Tue 5 Nov 12:38:12 UTC 2024 (1730810292)
RELEASE: latest (/lib/firmware/raspberrypi/bootloader-2712/latest)
Use raspi-config to change the release.

System

root@rpi5-usb:# cat /etc/rpi-issue
Raspberry Pi reference 2024-10-22
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, ecda2579ebbc125b321984c571e3128f6fd959d4, stage4
root@rpi5-usb:
# vcgencmd bootloader_version
2024/11/05 12:38:12
version 3c4fc886f1f3b34a794d7157e0f26ec5942408ca (release)
timestamp 1730810292
update-time 1730984545
capabilities 0x0000007f

root@rpi5-usb:# vcgencmd version
2024/11/05 12:38:12
Copyright (c) 2012 Broadcom
version 3c4fc886 (release) (embedded)
root@rpi5-usb:
# uname -a
Linux rpi5-usb 6.6.51+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64 GNU/Linux
root@rpi5-usb:~#

Bootloader logs

No response

USB boot

root@rpi5-usb:~# lsusb -vvv

Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 3.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 3
bMaxPacketSize0 9
idVendor 0x1d6b Linux Foundation
idProduct 0x0003 3.0 root hub
bcdDevice 6.06
iManufacturer 3 Linux 6.6.51+rpt-rpi-2712 xhci-hcd
iProduct 2 xHCI Host Controller
iSerial 1 xhci-hcd.1
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x001f
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 12
bMaxBurst 0
Hub Descriptor:
bLength 12
bDescriptorType 42
nNbrPorts 1
wHubCharacteristic 0x0009
Per-port power switching
Per-port overcurrent protection
bPwrOn2PwrGood 50 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
bHubDecLat 0.0 micro seconds
wHubDelay 0 nano seconds
DeviceRemovable 0x00
Hub Port Status:
Port 1: 0000.02a0 5Gbps power Rx.Detect
Binary Object Store Descriptor:
bLength 5
bDescriptorType 15
wTotalLength 0x000f
bNumDeviceCaps 1
SuperSpeed USB Device Capability:
bLength 10
bDescriptorType 16
bDevCapabilityType 3
bmAttributes 0x02
Latency Tolerance Messages (LTM) Supported
wSpeedsSupported 0x0008
Device can operate at SuperSpeed (5Gbps)
bFunctionalitySupport 1
Lowest fully-functional device speed is Full Speed (12Mbps)
bU1DevExitLat 10 micro seconds
bU2DevExitLat 2047 micro seconds
can't get debug descriptor: Resource temporarily unavailable
Device Status: 0x0001
Self Powered

Bus 003 Device 002: ID 346d:5678
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64
idVendor 0x346d
idProduct 0x5678
bcdDevice 2.00
iManufacturer 1
iProduct 2
iSerial 3 8649321074347787162
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0020
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0x80
(Bus Powered)
MaxPower 100mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 8 Mass Storage
bInterfaceSubClass 6 SCSI
bInterfaceProtocol 80 Bulk-Only
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 0
Device Qualifier (for other device speed):
bLength 10
bDescriptorType 6
bcdUSB 2.00
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64
bNumConfigurations 1
can't get debug descriptor: Resource temporarily unavailable
Device Status: 0x0000
(Bus Powered)

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 1 Single TT
bMaxPacketSize0 64
idVendor 0x1d6b Linux Foundation
idProduct 0x0002 2.0 root hub
bcdDevice 6.06
iManufacturer 3 Linux 6.6.51+rpt-rpi-2712 xhci-hcd
iProduct 2 xHCI Host Controller
iSerial 1 xhci-hcd.1
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 12
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 2
wHubCharacteristic 0x0009
Per-port power switching
Per-port overcurrent protection
TT think time 8 FS bits
bPwrOn2PwrGood 10 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
DeviceRemovable 0x00
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0100 power
Port 2: 0000.0503 highspeed power enable connect
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
Device Status: 0x0001
Self Powered

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 3.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 3
bMaxPacketSize0 9
idVendor 0x1d6b Linux Foundation
idProduct 0x0003 3.0 root hub
bcdDevice 6.06
iManufacturer 3 Linux 6.6.51+rpt-rpi-2712 xhci-hcd
iProduct 2 xHCI Host Controller
iSerial 1 xhci-hcd.0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x001f
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 12
bMaxBurst 0
Hub Descriptor:
bLength 12
bDescriptorType 42
nNbrPorts 1
wHubCharacteristic 0x0009
Per-port power switching
Per-port overcurrent protection
bPwrOn2PwrGood 50 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
bHubDecLat 0.0 micro seconds
wHubDelay 0 nano seconds
DeviceRemovable 0x00
Hub Port Status:
Port 1: 0000.02a0 5Gbps power Rx.Detect
Binary Object Store Descriptor:
bLength 5
bDescriptorType 15
wTotalLength 0x000f
bNumDeviceCaps 1
SuperSpeed USB Device Capability:
bLength 10
bDescriptorType 16
bDevCapabilityType 3
bmAttributes 0x02
Latency Tolerance Messages (LTM) Supported
wSpeedsSupported 0x0008
Device can operate at SuperSpeed (5Gbps)
bFunctionalitySupport 1
Lowest fully-functional device speed is Full Speed (12Mbps)
bU1DevExitLat 10 micro seconds
bU2DevExitLat 2047 micro seconds
can't get debug descriptor: Resource temporarily unavailable
Device Status: 0x0001
Self Powered

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 1 Single TT
bMaxPacketSize0 64
idVendor 0x1d6b Linux Foundation
idProduct 0x0002 2.0 root hub
bcdDevice 6.06
iManufacturer 3 Linux 6.6.51+rpt-rpi-2712 xhci-hcd
iProduct 2 xHCI Host Controller
iSerial 1 xhci-hcd.0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 12
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 2
wHubCharacteristic 0x0009
Per-port power switching
Per-port overcurrent protection
TT think time 8 FS bits
bPwrOn2PwrGood 10 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
DeviceRemovable 0x00
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0100 power
Port 2: 0000.0100 power
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
Device Status: 0x0001
Self Powered
root@rpi5-usb:~#

NVMe boot

Node Generic SN Model Namespace Usage Format FW Rev


/dev/nvme0n1 /dev/ng0n1 799PE0N0PQEN KBG40ZNS256G NVMe TOSHIBA 256GB 1 256.06 GB / 256.06 GB 4 KiB + 0 B 10410106

root@rpi5-usb:~# nvme id-ctrl -H /dev/nvme0
NVME Identify Controller:
vid : 0x1e0f
ssvid : 0x1e0f
sn : 799PE0N0PQEN
mn : KBG40ZNS256G NVMe TOSHIBA 256GB
fr : 10410106
rab : 3
ieee : 8ce38e
cmic : 0
[3:3] : 0 ANA not supported
[2:2] : 0 PCI
[1:1] : 0 Single Controller
[0:0] : 0 Single Port

mdts : 9
cntlid : 0
ver : 0x10300
rtd3r : 0x7a120
rtd3e : 0x2bf20
oaes : 0x200
[31:31] : 0 Discovery Log Change Notice Not Supported
[27:27] : 0 Zone Descriptor Changed Notices Not Supported
[15:15] : 0 Normal NSS Shutdown Event Not Supported
[14:14] : 0 Endurance Group Event Aggregate Log Page Change Notice Not Supported
[13:13] : 0 LBA Status Information Notices Not Supported
[12:12] : 0 Predictable Latency Event Aggregate Log Change Notices Not Supported
[11:11] : 0 Asymmetric Namespace Access Change Notices Not Supported
[9:9] : 0x1 Firmware Activation Notices Supported
[8:8] : 0 Namespace Attribute Changed Event Not Supported

ctratt : 0x2
[19:19] : 0 Flexible Data Placement Not Supported
[15:15] : 0 Extended LBA Formats Not Supported
[14:14] : 0 Delete NVM Set Not Supported
[13:13] : 0 Delete Endurance Group Not Supported
[12:12] : 0 Variable Capacity Management Not Supported
[11:11] : 0 Fixed Capacity Management Not Supported
[10:10] : 0 Multi Domain Subsystem Not Supported
[9:9] : 0 UUID List Not Supported
[8:8] : 0 SQ Associations Not Supported
[7:7] : 0 Namespace Granularity Not Supported
[6:6] : 0 Traffic Based Keep Alive Not Supported
[5:5] : 0 Predictable Latency Mode Not Supported
[4:4] : 0 Endurance Groups Not Supported
[3:3] : 0 Read Recovery Levels Not Supported
[2:2] : 0 NVM Sets Not Supported
[1:1] : 0x1 Non-Operational Power State Permissive Supported
[0:0] : 0 128-bit Host Identifier Not Supported

rrls : 0
cntrltype : 0
[7:2] : 0 Reserved
[1:0] : 0 Controller type not reported
fguid : 00000000-0000-0000-0000-000000000000
crdt1 : 0
crdt2 : 0
crdt3 : 0
nvmsr : 0
[1:1] : 0 NVM subsystem Not part of an Enclosure
[0:0] : 0 NVM subsystem Not part of an Storage Device

vwci : 0
[7:7] : 0 VPD Write Cycles Remaining field is Not valid.
[6:0] : 0 VPD Write Cycles Remaining

mec : 0
[1:1] : 0 NVM subsystem Not contains a Management Endpoint on a PCIe port
[0:0] : 0 NVM subsystem Not contains a Management Endpoint on an SMBus/I2C port

oacs : 0x17
[10:10] : 0 Lockdown Command and Feature Not Supported
[9:9] : 0 Get LBA Status Capability Not Supported
[8:8] : 0 Doorbell Buffer Config Not Supported
[7:7] : 0 Virtualization Management Not Supported
[6:6] : 0 NVMe-MI Send and Receive Not Supported
[5:5] : 0 Directives Not Supported
[4:4] : 0x1 Device Self-test Supported
[3:3] : 0 NS Management and Attachment Not Supported
[2:2] : 0x1 FW Commit and Download Supported
[1:1] : 0x1 Format NVM Supported
[0:0] : 0x1 Security Send and Receive Supported

acl : 3
aerl : 7
frmw : 0x14
[5:5] : 0 Multiple FW or Boot Update Detection Not Supported
[4:4] : 0x1 Firmware Activate Without Reset Supported
[3:1] : 0x2 Number of Firmware Slots
[0:0] : 0 Firmware Slot 1 Read/Write

lpa : 0xe
[6:6] : 0 Telemetry Log Data Area 4 Not Supported
[5:5] : 0 LID 0x0, Scope of each command in LID 0x5, 0x12, 0x13 Not Supported
[4:4] : 0 Persistent Event log Not Supported
[3:3] : 0x1 Telemetry host/controller initiated log page Supported
[2:2] : 0x1 Extended data for Get Log Page Supported
[1:1] : 0x1 Command Effects Log Page Supported
[0:0] : 0 SMART/Health Log Page per NS Not Supported

elpe : 255
[7:0] : 255 (0's based) Error Log Page Entries (ELPE)

npss : 4
[7:0] : 4 (0's based) Number of Power States Support (NPSS)

avscc : 0
[0:0] : 0 Admin Vendor Specific Commands uses Vendor Specific Format

apsta : 0x1
[0:0] : 0x1 Autonomous Power State Transitions Supported

wctemp : 356
[15:0] : 83°C (356 Kelvin) Warning Composite Temperature Threshold (WCTEMP)

cctemp : 360
[15:0] : 87°C (360 Kelvin) Critical Composite Temperature Threshold (CCTEMP)

mtfa : 0
hmpre : 15616
hmmin : 5888
tnvmcap : 256,060,514,304
[127:0] : 256,060,514,304
Total NVM Capacity (TNVMCAP)

unvmcap : 0
[127:0] : 0
Unallocated NVM Capacity (UNVMCAP)

rpmbs : 0
[31:24]: 0 Access Size
[23:16]: 0 Total Size
[5:3] : 0 Authentication Method
[2:0] : 0 Number of RPMB Units

edstt : 11
dsto : 1
fwug : 1
kas : 0
hctma : 0x1
[0:0] : 0x1 Host Controlled Thermal Management Supported

mntmt : 0
[15:0] : -273°C (0 Kelvin) Minimum Thermal Management Temperature (MNTMT)

mxtmt : 356
[15:0] : 83°C (356 Kelvin) Maximum Thermal Management Temperature (MXTMT)

sanicap : 0x2
[31:30] : 0 Additional media modification after sanitize operation completes successfully is not defined
[29:29] : 0 No-Deallocate After Sanitize bit in Sanitize command Supported
[2:2] : 0 Overwrite Sanitize Operation Not Supported
[1:1] : 0x1 Block Erase Sanitize Operation Supported
[0:0] : 0 Crypto Erase Sanitize Operation Not Supported

hmminds : 0
hmmaxd : 0
nsetidmax : 0
endgidmax : 0
anatt : 0
anacap : 0
[7:7] : 0 Non-zero group ID Not Supported
[6:6] : 0 Group ID does change
[4:4] : 0 ANA Change state Not Supported
[3:3] : 0 ANA Persistent Loss state Not Supported
[2:2] : 0 ANA Inaccessible state Not Supported
[1:1] : 0 ANA Non-optimized state Not Supported
[0:0] : 0 ANA Optimized state Not Supported

anagrpmax : 0
nanagrpid : 0
pels : 0
domainid : 0
megcap : 0
sqes : 0x66
[7:4] : 0x6 Max SQ Entry Size (64)
[3:0] : 0x6 Min SQ Entry Size (64)

cqes : 0x44
[7:4] : 0x4 Max CQ Entry Size (16)
[3:0] : 0x4 Min CQ Entry Size (16)

maxcmd : 0
nn : 1
oncs : 0x5f
[8:8] : 0 Copy Not Supported
[7:7] : 0 Verify Not Supported
[6:6] : 0x1 Timestamp Supported
[5:5] : 0 Reservations Not Supported
[4:4] : 0x1 Save and Select Supported
[3:3] : 0x1 Write Zeroes Supported
[2:2] : 0x1 Data Set Management Supported
[1:1] : 0x1 Write Uncorrectable Supported
[0:0] : 0x1 Compare Supported

fuses : 0
[0:0] : 0 Fused Compare and Write Not Supported

fna : 0
[3:3] : 0 Format NVM Broadcast NSID (FFFFFFFFh) Supported
[2:2] : 0 Crypto Erase Not Supported as part of Secure Erase
[1:1] : 0 Crypto Erase Applies to Single Namespace(s)
[0:0] : 0 Format Applies to Single Namespace(s)

vwc : 0x1
[2:1] : 0 Support for the NSID field set to FFFFFFFFh is not indicated
[0:0] : 0x1 Volatile Write Cache Present

awun : 65535
awupf : 0
icsvscc : 1
[0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format

nwpc : 0
[2:2] : 0 Permanent Write Protect Not Supported
[1:1] : 0 Write Protect Until Power Supply Not Supported
[0:0] : 0 No Write Protect and Write Protect Namespace Not Supported

acwu : 0
ocfs : 0
[1:1] : 0 Controller Copy Format 1h Not Supported
[0:0] : 0 Controller Copy Format 0h Not Supported

sgls : 0
[15:8] : 0 SGL Descriptor Threshold
[1:0] : 0 Scatter-Gather Lists Not Supported

mnan : 0
maxdna : 0
maxcna : 0
subnqn : nqn.2018-06.com.toshiba-memory:KBG40ZNS256G NVMe TOSHIBA 256GB:799PE0N0PQEN
ioccsz : 0
iorcsz : 0
icdoff : 0
fcatt : 0
[0:0] : 0 Dynamic Controller Model

msdbd : 0
ofcs : 0
[0:0] : 0 Disconnect command Not Supported

ps 0 : mp:3.60W operational enlat:1 exlat:1 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-
active_power_workload:-
ps 1 : mp:2.60W operational enlat:1 exlat:1 rrt:1 rrl:1
rwt:1 rwl:1 idle_power:- active_power:-
active_power_workload:-
ps 2 : mp:2.20W operational enlat:1 exlat:1 rrt:2 rrl:2
rwt:2 rwl:2 idle_power:- active_power:-
active_power_workload:-
ps 3 : mp:0.0500W non-operational enlat:800 exlat:1200 rrt:3 rrl:3
rwt:3 rwl:3 idle_power:- active_power:-
active_power_workload:-
ps 4 : mp:0.0050W non-operational enlat:3000 exlat:32000 rrt:4 rrl:4
rwt:4 rwl:4 idle_power:- active_power:-
active_power_workload:-
root@rpi5-usb:~#

root@rpi5-usb:~# nvme list-ns /dev/nvme0
[ 0]:0x1

root@rpi5-usb:~# nvme id-ns -H /dev/nvme0 --namespace-id=1
NVME Identify Namespace 1:
nsze : 0x3b9e656
ncap : 0x3b9e656
nuse : 0x3b9e656
nsfeat : 0
[4:4] : 0 NPWG, NPWA, NPDG, NPDA, and NOWS are Not Supported
[3:3] : 0 NGUID and EUI64 fields if non-zero, Reused
[2:2] : 0 Deallocated or Unwritten Logical Block error Not Supported
[1:1] : 0 Namespace uses AWUN, AWUPF, and ACWU
[0:0] : 0 Thin Provisioning Not Supported

nlbaf : 1
flbas : 0x1
[6:5] : 0 Most significant 2 bits of Current LBA Format Selected
[4:4] : 0 Metadata Transferred in Separate Contiguous Buffer
[3:0] : 0x1 Least significant 4 bits of Current LBA Format Selected

mc : 0
[1:1] : 0 Metadata Pointer Not Supported
[0:0] : 0 Metadata as Part of Extended Data LBA Not Supported

dpc : 0
[4:4] : 0 Protection Information Transferred as Last 8 Bytes of Metadata Not Supported
[3:3] : 0 Protection Information Transferred as First 8 Bytes of Metadata Not Supported
[2:2] : 0 Protection Information Type 3 Not Supported
[1:1] : 0 Protection Information Type 2 Not Supported
[0:0] : 0 Protection Information Type 1 Not Supported

dps : 0
[3:3] : 0 Protection Information is Transferred as Last 8 Bytes of Metadata
[2:0] : 0 Protection Information Disabled

nmic : 0
[0:0] : 0 Namespace Multipath Not Capable

rescap : 0
[7:7] : 0 Ignore Existing Key - Used as defined in revision 1.2.1 or earlier
[6:6] : 0 Exclusive Access - All Registrants Not Supported
[5:5] : 0 Write Exclusive - All Registrants Not Supported
[4:4] : 0 Exclusive Access - Registrants Only Not Supported
[3:3] : 0 Write Exclusive - Registrants Only Not Supported
[2:2] : 0 Exclusive Access Not Supported
[1:1] : 0 Write Exclusive Not Supported
[0:0] : 0 Persist Through Power Loss Not Supported

fpi : 0
[7:7] : 0 Format Progress Indicator Not Supported

dlfeat : 1
[4:4] : 0 Guard Field of Deallocated Logical Blocks is set to 0xFFFF
[3:3] : 0 Deallocate Bit in the Write Zeroes Command is Not Supported
[2:0] : 0x1 Bytes Read From a Deallocated Logical Block and its Metadata are 0x00

nawun : 0
nawupf : 0
nacwu : 0
nabsn : 0
nabo : 0
nabspf : 0
noiob : 0
nvmcap : 256,060,514,304
mssrl : 0
mcl : 0
msrc : 0
nulbaf : 0
anagrpid: 0
nsattr : 0
nvmsetid: 0
endgid : 0
nguid : 01000000000000008ce38e040011a073
eui64 : 8ce38e040011a073
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x3 Degraded
LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better (in use)
root@rpi5-usb:~#

Network (TFTP boot)

No response

@peterharperuk
Copy link
Collaborator

My guess is that there's something weird with your boot partition on the NVME drive. Why is it 4G in size? Are the cmdline.txt files the same on the USB disk and the NVME disk? I boot my Pi 5 from NVME all the time, so you've probably done something wrong.

@doubleuerner
Copy link
Author

well, if i just download the img.xz and unpack it and use dd to write it to NVMe disk, then i get this 4G boot partition. I discovered that dd used 512 and the nvme 4096 blocksize. So then i looked with fdisk into the img-file and see a partition of 512M, and that makes sense because the bs = x8, so 4GB. so i did all by hand using parted/fdisk and mounted and copied all files. Then still i doesnt work, and used the rpi-imager: there the 4GB boot partition was there , no succes. Then used sd card copier, and eventually no succes, until i also used the SUB-MSB stick. And yes the content on NVMe is exact the same as USB. so also cmdline.txt.

@peterharperuk
Copy link
Collaborator

We don't support 4096 byte block size in the bootloader. That's the problem.

@doubleuerner
Copy link
Author

doubleuerner commented Nov 7, 2024

Aha, okay then I go change the sector size to 512 and try to dd the img to nvmen1. Thanks for the hint! I'll let you know the result!

https://man.archlinux.org/man/nvme-format.1.en

@doubleuerner
Copy link
Author

It works!

Thanks , i learned a lot having tried 'everything' and taking all kinds of directions, but finally it works and i understand why.

@peterharperuk
Copy link
Collaborator

One day we might fix this. It would be interesting to know if performance is noticeably better when the linux partition uses a 4096 byte block size.

@peterharperuk
Copy link
Collaborator

Actually, reopen. We should probably throw a more obvious error to stop people wasting their time with this in future?

@peterharperuk peterharperuk self-assigned this Nov 7, 2024
@doubleuerner
Copy link
Author

doubleuerner commented Nov 7, 2024

Well, I think this struggle appearing to be sector size other than 512 is very common for NVMe's If I just Google on this you see discussion about turning to a different block size in general. The NVMe card i have was from an old windows machine. I think , me as an hobbyist as many others re-use stuff and try to make it work, i don't know if you buy a new NVMe the default is 512 of something else. I thing even the rpi-imager uses dd as underlying method and i know this turns into weird partition table and filesystem not being mount-able.

@doubleuerner
Copy link
Author

doubleuerner commented Nov 7, 2024

The M.2 HAT+ has documentation explaining it all, but not this exception of 4096 sector size which is not able to boot.

@peterharperuk
Copy link
Collaborator

Yeah, even I missed "nvme: error 10" in your screen grab which means NMVE_ERROR_UNSUPPORTED_BLOCK_SIZE

@timg236
Copy link
Collaborator

timg236 commented Nov 7, 2024

The feature is on the todo list (but on the back burner) - to make this useful it will also require updates to RPi Imager / RPi OS for provisioning images in this format.

raspberrypi/firmware#1840

@peterharperuk
Copy link
Collaborator

I'm just suggesting making the error more obvious for now.

@doubleuerner
Copy link
Author

Yeah, even I missed "nvme: error 10" in your screen grab which means NMVE_ERROR_UNSUPPORTED_BLOCK_SIZE

google on nvme: error 10 yesterday didn't show up something, also i tried to see if the source code is somewhere. But is this something that can be found by me for instance, or is it something in the code nobody except the developers can find?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants