RK3566 port #6954

DaniilKl · 2024-07-20T15:34:43Z

This port is a part of integration of safety features into Zarhus project, the main effort can be found in the following PR: zarhus/meta-zarhus-bsp-rockchip#5, so the precise hardware and software configuration being used can be checked there. Though the project uses Yocto Project as a build environment, all the test binaries are built manually using several tools, so the configuration and changes in the PR may not be up to date.

Brief description of test environment.

Hardware:

Radxa CM3 SoM based on RK3566 SoC;
Radxa CM3 IO extention board.

Software:

I think this image explains software stack in the best way:

Source: opensource.rock-chips.com.

The red boot flow is being used for tests.

This is my first interaction with OPTEE and Trust Zone stuff, so I will be grateful for any help here!

DaniilKl · 2024-07-20T16:07:17Z

For now, I was able to boot the describett boot flow up to Linux userspace without any errors/warning from OPTEE side:

I/TC: OP-TEE version: 4.1.0-dev (gcc version 13.2.0 (GCC)) #1 Fri Jan 19 17:14:14 UTC 2024 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: Primary CPU switching to normal world boot
INFO:    BL31: BL32 initialization succeded!
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0xa00000
INFO:    SPSR = 0x3c9

Further steps and problems to solve

Linking problem

After build I get an ELF with segments that seem to be ok for my configuration:

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000010000 0x0000000008400000 0x0000000008400000
                 0x000000000006ee98 0x00000000000ac400  RWE    0x10000
  LOAD           0x0000000000080000 0x0000000008600000 0x0000000008600000
                 0x0000000000005f98 0x0000000000005f98  RW     0x10000
  DYNAMIC        0x0000000000080000 0x0000000008600000 0x0000000008600000
                 0x0000000000000110 0x0000000000000110  RW     0x8
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10

The ELF get linked with U-Boot and ATF into one image and then being booted. But during boot ATF cannot boot OPTEE (it gets blocked for several second on communicate INFO: BL31: Initializing BL32, drops it, and continues to boot U-Boot). After some debugging I found out that SPL images gives to ATF 0x8600000 instead of 0x8400000 as an TEE entry point. After some digging I was able to relink the image manually so the entry was 0x840000 and boot it (the lopgs from the beginning). Still searching of what I have done wrong here.

Linux Kernel oops

After integrating self built OPTEE and ATF I got following oops during Linux Kernel boot:

[    0.493196] scmi_protocol scmi_dev.1: Enabled polling mode TX channel - prot_id:16
[    0.494329] arm-scmi firmware:scmi: SCMI Notifications - Core Enabled.
[    0.505441] mmc_host mmc1: Bus speed (slot 0) = 375000Hz (slot req 400000Hz, actual 375000HZ div = 0)
[    0.554984] ------------[ cut here ]------------
[    0.555414] WARNING: CPU: 2 PID: 1 at /drivers/firmware/arm_scmi/shmem.c:60 shmem_tx_prepare+0xe4/0x150
[    0.556277] Modules linked in:
[    0.556569] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 6.6.23-yocto-standard-00118-g2d01bc1d4eea #1
[    0.557374] Hardware name: Radxa Compute Module 3(CM3) IO Board (DT)
[    0.557946] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.558577] pc : shmem_tx_prepare+0xe4/0x150
[    0.558971] lr : shmem_tx_prepare+0x124/0x150
[    0.559372] sp : ffff80008169b910
[    0.559676] x29: ffff80008169b910 x28: 0000000000000000 x27: ffff0000013ae7a8
[    0.560333] x26: ffff000001294080 x25: ffff8000815bc3d8 x24: 0000000000000000
[    0.560986] x23: 0000000000000000 x22: ffff800081875004 x21: ffff0000013a8a40
[    0.561638] x20: ffff800081875000 x19: ffff00000133d880 x18: 0000000000000006
[    0.562288] x17: 746f7270202d206c x16: 656e6e6168632058 x15: 076e076f07690774
[    0.562941] x14: 0761076307690766 x13: ffff8000814b0438 x12: 00000000000002dc
[    0.563592] x11: 0000000000000040 x10: ffff000001296138 x9 : ffff000001296130
[    0.564244] x8 : ffff0000009d7dc0 x7 : 0000000000000000 x6 : 0000000007d7939b
[    0.564896] x5 : 00ffffffffffffff x4 : 00206cc800324b00 x3 : 0000000000000018
[    0.565547] x2 : 0000000000000112 x1 : 0000000000000000 x0 : 0000000000000000
[    0.566198] Call trace:
[    0.566428]  shmem_tx_prepare+0xe4/0x150
[    0.566794]  smc_send_message+0x44/0xe4
[    0.567152]  do_xfer+0xb0/0x1b4
[    0.567451]  version_get+0x90/0xbc
[    0.567770]  scmi_base_protocol_init+0x5c/0x458
[    0.568190]  scmi_get_protocol_instance+0x160/0x28c
[    0.568639]  scmi_probe+0x2b0/0x720
[    0.568967]  platform_probe+0x68/0xc8
[    0.569315]  really_probe+0x148/0x2b0
[    0.569657]  __driver_probe_device+0x78/0x12c
[    0.570060]  driver_probe_device+0x3c/0x15c
[    0.570446]  __driver_attach+0x90/0x19c
[    0.570802]  bus_for_each_dev+0x7c/0xdc
[    0.571159]  driver_attach+0x24/0x30
[    0.571493]  bus_add_driver+0xe8/0x1f8
[    0.571841]  driver_register+0x5c/0x124
[    0.572198]  __platform_driver_register+0x28/0x34
[    0.572635]  scmi_driver_init+0x80/0xbc
[    0.572998]  do_one_initcall+0x80/0x1c8
[    0.573355]  kernel_init_freeable+0x1bc/0x284
[    0.573760]  kernel_init+0x24/0x1dc
[    0.574087]  ret_from_fork+0x10/0x20
[    0.574421] ---[ end trace 0000000000000000 ]---
[    0.574874] scmi_protocol scmi_dev.1: Timeout waiting for a free TX channel !
[    0.575544] arm-scmi firmware:scmi: unable to communicate with SCMI
[    0.576325] arm-scmi: probe of firmware:scmi failed with error -95
[    0.577366] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[    0.578660] usbcore: registered new interface driver usbhid
[    0.579173] usbhid: USB HID core driver
[    0.582295] hw perfevents: enabled with armv8_cortex_a55 PMU driver, 7 counters available
[    0.587587] NET: Registered PF_PACKET protocol family
[    0.588129] 9pnet: Installing 9P2000 support
[    0.588465] mmc_host mmc1: Bus speed (slot 0) = 50000000Hz (slot req 50000000Hz, actual 50000000HZ div = 0)
[    0.589551] mmc1: new high speed SDHC card at address aaaa
[    0.591110] mmcblk1: mmc1:aaaa SC16G 14.8 GiB

For now I do not know exactly what may be the cause, but, I suppose, it is OPTEE and memory initialization related stuff. The code in core/arch/arm/plat-rockchip/platform_rk3566.c was copied from core/arch/arm/plat-rockchip/platform_rk3566.c without changes, with assumption that both SoC's have the same Armv8 architecture and therefore memory initialization will not need a lot of changes. This means that I do not fully understand the code there.

But, actually, the problem might as well has come from ATF, Because I am using latest master of trusted-firmware-a to build an ATF binary because only has support for RK3566, and master builds may be unstable.

DaniilKl · 2024-07-20T16:14:19Z

core/arch/arm/plat-rockchip/platform_rk3566.c

+#include <platform.h>
+#include <platform_config.h>
+
+#define SGRF_DDRRGN_CON0_16(n)		((n) * 4)


And this one I do not understand. This seems to be some RK3399 registers writing, but I have not been able do understand what exactly is going on here so far.

This seems to be a helper macro for DDR region registers as stated here. And there is a precise address of the SGRF registers for rk3399, but I have not been able to find these for rk3566 or rk3568 so far 😢 .

jenswi-linaro

This port is a bit shaky, obviously since it crashes in the kernel but also that it's based on unconfirmed assumptions.

jenswi-linaro · 2024-07-26T06:19:40Z

core/mm/core_mmu.c

@@ -2418,10 +2418,19 @@ void *phys_to_virt(paddr_t pa, enum teecore_memtypes m, size_t len)
 		va = NULL;
 		break;
 	default:
+		if ( m == MEM_AREA_IO_SEC )


Please remove the changes in this file, it looks like it's only debug code while you're testing things.

I will do it when submitting this PR to the final review. Until then it will be in draft state and I will keep here some WIP or debug commits.

jenswi-linaro · 2024-07-26T06:22:16Z

core/arch/arm/plat-rockchip/platform_rk3566.c

@@ -0,0 +1,44 @@
+#include <common.h>


This file looks like a copy of platform_rk3399.c. I'd rather reuse the same code than use diverging copies that are poorly understood.

Sure, but firstly I need to get SGRF info from Rockchip.

Unfortunately, RockChip will not provide any SGRF to anyone who is not a $$$ customer. I have a somewhat decent relationship with them, I even reported to them a critical security vulnerability with one of their SoCs and they still won't help much, even signing NDAs.

jenswi-linaro · 2024-07-26T06:22:44Z

core/arch/arm/plat-rockchip/platform_config.h

@@ -66,6 +66,34 @@
 #define SGRF_BASE		(MMIO_BASE + 0x07330000)
 #define SGRF_SIZE		SIZE_K(64)

+#elif defined(PLATFORM_FLAVOR_rk3566)
+
+/* This information is from Technical Reference Manual for RK3568, which is said


/* * Please format * multiline comments * like this. */

jenswi-linaro · 2024-07-26T06:26:28Z

core/arch/arm/plat-rockchip/main.c

@@ -24,7 +24,11 @@ register_phys_mem_pgdir(MEM_AREA_IO_SEC, GIC_BASE, GIC_SIZE);

 void boot_primary_init_intc(void)
 {
+	#ifdef PLATFORM_FLAVOR_rk3566


Please check for GICR_BASE or if CFG_ARM_GICV3 is enabled.
You don't need to indent preprocessor directives.

jenswi-linaro · 2024-07-26T06:30:49Z

core/arch/arm/plat-rockchip/conf.mk

+arm32-platform-cpuarch  := cortex-a55
+
+$(call force,CFG_TEE_CORE_NB_CORE,4)
+# cortex-a55 sopports GICv3 or GICv4. There is no support for GICv4 in OP-TEE,


What does the soc have?

According to rk3568 TRF the SoC has GIC-600, which implements GICv3.

DaniilKl · 2024-07-26T09:48:18Z

Currently this port is not ready for review (so, currently in draft status), the cause is: missing SGRF info for RK3566. I am currently waiting for a response from Rockchip, so, I hope I will get needed information soon.

Signed-off-by: Daniil Klimuk <[email protected]>

DaniilKl · 2024-09-06T10:31:54Z

Some rebasing on new master.

DaniilKl · 2024-09-25T12:30:50Z

Current port status

It boots

(...)
INFO:    BL31: Initializing runtime services
INFO:    BL31: Initializing BL32
I/TC: 
I/TC: No non-secure external DT
I/TC: OP-TEE version: 4.1.0-dev (gcc version 13.2.0 (GCC)) #1 Fri Jan 19 17:14:14 UTC 2024 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: Primary CPU switching to normal world boot
INFO:    BL31: Preparing for EL3 exit to normal world
(...)
root@quartz64-a:~# dmesg
(...)
[    0.014933] EFI services will not be available.
[    0.015857] smp: Bringing up secondary CPUs ...
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
I/TC: Secondary CPU 2 initializing
I/TC: Secondary CPU 2 switching to normal world boot
I/TC: Secondary CPU 3 initializing
I/TC: Secondary CPU 3 switching to normal world boot
[    0.018001] Detected VIPT I-cache on CPU1
[    0.018085] GICv3: CPU1: found redistributor 100 region 0:0x00000000fd480000
(...)
[    0.393425] hw perfevents: enabled with armv8_cortex_a55 PMU driver, 7 counters available
[    0.396831] optee: probing for conduit method.
I/TC: Reserved shared memory is enabled
I/TC: Dynamic shared memory is disabled
I/TC: Normal World virtualization support is disabled
I/TC: Asynchronous notifications are disabled
[    0.397282] optee: revision 4.1 (18b424c2)
[    0.399550] optee: initialized driver
(...)

It seems to work

root@quartz64-a:~# p11 --show-info
I/TC: WARNING (insecure configuration): Failed to get monotonic counter for REE FS, using 0
I/TC: WARNING (insecure configuration): Failed to commit dirh counter 2
Cryptoki version 2.40
Manufacturer     Linaro
Library          OP-TEE PKCS11 Cryptoki library (ver 0.1)
Using slot 0 with a present token (0x0)
root@quartz64-a:~# p11 --list-slots 
Available slots:
Slot 0 (0x0): OP-TEE PKCS11 TA - TEE UUID 94e9ab89-4c43-56ea-8b35-45dc07226830
  token state:   uninitialized
Slot 1 (0x1): OP-TEE PKCS11 TA - TEE UUID 94e9ab89-4c43-56ea-8b35-45dc07226830
  token state:   uninitialized
Slot 2 (0x2): OP-TEE PKCS11 TA - TEE UUID 94e9ab89-4c43-56ea-8b35-45dc07226830
  token state:   uninitialized
root@quartz64-a:~# p11 --init-token --label mytoken --so-pin 1234567890
Using slot 0 with a present token (0x0)
Token successfully initialized
root@quartz64-a:~# p11 --label mytoken --login --so-pin 1234567890 --init-pin --pin 12345
Using slot 0 with a present token (0x0)
User PIN successfully initialized
root@quartz64-a:~# p11 --list-slots
Available slots:
Slot 0 (0x0): OP-TEE PKCS11 TA - TEE UUID 94e9ab89-4c43-56ea-8b35-45dc07226830
  token label        : mytoken
  token manufacturer : Linaro
  token model        : OP-TEE TA
  token flags        : login required, rng, token initialized, PIN initialized
  hardware version   : 0.0
  firmware version   : 0.1
  serial num         : 0000000000000000
  pin min/max        : 4/128
Slot 1 (0x1): OP-TEE PKCS11 TA - TEE UUID 94e9ab89-4c43-56ea-8b35-45dc07226830
  token state:   uninitialized
Slot 2 (0x2): OP-TEE PKCS11 TA - TEE UUID 94e9ab89-4c43-56ea-8b35-45dc07226830
  token state:   uninitialized
root@quartz64-a:~# p11 --list-mechanisms                                                                              
Using slot 0 with a present token (0x0)
Supported mechanisms:
  SHA224-RSA-PKCS-PSS, keySize={256,4096}, sign, verify
  SHA224-RSA-PKCS, keySize={256,4096}, sign, verify
  (...)
  AES-CBC, keySize={16,32}, encrypt, decrypt, wrap, unwrap
  AES-ECB, keySize={16,32}, encrypt, decrypt, wrap, unwrap
root@quartz64-a:~# p11 -l --pin 12345 --keypairgen --key-type EC:prime256v1 --label mykey
Using slot 0 with a present token (0x0)
Key pair generated:
Private Key Object; EC
  label:      mykey
  Usage:      sign, derive
  Access:     sensitive, always sensitive, never extractable, local
Public Key Object; EC  EC_POINT 256 bits
  EC_POINT:   044104c721ebadc9bf519efaa0d37ec0d1258986838dc54ea4beb4956a25d20a33d587e861850f9fc357181423a81ebdd54b952a
  EC_PARAMS:  06082a8648ce3d030107 (OID 1.2.840.10045.3.1.7)
  label:      mykey
  Usage:      verify, derive
  Access:     local

The problems and solutions

Kernel panic

The Linux Kernel panic was caused by absence of the OP-TEE OS memory nodes in Linux devicetree, so the following nodes fixed the problem:

    reserved-memory {
		#address-cells = <2>;
		#size-cells = <2>;
		ranges;

		optee@CFG_TZDRAM_START {
			reg = <CFG_TZDRAM_START CFG_TZDRAM_SIZE>;
            no-map;
		};

		optee_shm: optee_shared_mem@CFG_SHMEM_START {
			reg = <CFG_SHMEM_START CFG_SHMEM_SIZE>;
		};
	};

	firmware {
		optee: optee {
			compatible = "linaro,optee-tz";
			method = "smc";
			shm = <&optee_shm>;
		};
    };

MMU problem

I have not been able to get info about MMU registers from Rockchip yet. So, though the port seems to be working, it cannot be finished. I will try to do smth here, but I have a bad feeling about this.

github-actions · 2024-10-26T01:50:13Z

This pull request has been marked as a stale pull request because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment, otherwise this pull request will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.

DaniilKl · 2024-10-26T16:14:04Z

The communication with Rockchip is still ongoing.

klogg · 2024-11-13T08:27:03Z

Hey @DaniilKl, any news from Rockchip?

DaniilKl · 2024-11-13T09:05:40Z

Hey @klogg, unfortunately no. And I am a bit pessimistic about it.

DaniilKl marked this pull request as draft July 20, 2024 15:35

DaniilKl changed the title ~~EK3566 port~~ RK3566 port Jul 20, 2024

DaniilKl commented Jul 20, 2024

View reviewed changes

jenswi-linaro reviewed Jul 26, 2024

View reviewed changes

DaniilKl added 2 commits September 6, 2024 12:28

core: arch: arm: plat-rockchip: add rk3566 support

4a06169

Signed-off-by: Daniil Klimuk <[email protected]>

debug patch

181dbb4

Signed-off-by: Daniil Klimuk <[email protected]>

DaniilKl force-pushed the rk3566-port branch from a7f277b to 181dbb4 Compare September 6, 2024 10:29

github-actions bot added the Stale label Oct 26, 2024

github-actions bot removed the Stale label Oct 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RK3566 port #6954

RK3566 port #6954

DaniilKl commented Jul 20, 2024

DaniilKl commented Jul 20, 2024

DaniilKl Jul 20, 2024

DaniilKl Jul 23, 2024

jenswi-linaro left a comment

jenswi-linaro Jul 26, 2024

DaniilKl Jul 26, 2024

jenswi-linaro Jul 26, 2024

DaniilKl Jul 26, 2024

DualTachyon Nov 11, 2024

jenswi-linaro Jul 26, 2024

jenswi-linaro Jul 26, 2024

jenswi-linaro Jul 26, 2024

DaniilKl Jul 26, 2024

DaniilKl commented Jul 26, 2024 •

edited

Loading

DaniilKl commented Sep 6, 2024

DaniilKl commented Sep 25, 2024

github-actions bot commented Oct 26, 2024

DaniilKl commented Oct 26, 2024

klogg commented Nov 13, 2024

DaniilKl commented Nov 13, 2024

RK3566 port #6954

Are you sure you want to change the base?

RK3566 port #6954

Conversation

DaniilKl commented Jul 20, 2024

DaniilKl commented Jul 20, 2024

Further steps and problems to solve

Linking problem

Linux Kernel oops

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jenswi-linaro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DaniilKl commented Jul 26, 2024 • edited Loading

DaniilKl commented Sep 6, 2024

DaniilKl commented Sep 25, 2024

Current port status

It boots

It seems to work

The problems and solutions

Kernel panic

MMU problem

github-actions bot commented Oct 26, 2024

DaniilKl commented Oct 26, 2024

klogg commented Nov 13, 2024

DaniilKl commented Nov 13, 2024

DaniilKl commented Jul 26, 2024 •

edited

Loading