Apache NuttX RTOS on PinePhone

Read the articles...

• "Apache NuttX RTOS on Arm Cortex-A53: How it might run on PinePhone"

• "PinePhone boots Apache NuttX RTOS"

• "NuttX RTOS on PinePhone: Fixing the Interrupts"

Apache NuttX RTOS now runs on Arm Cortex-A53 with Multi-Core SMP...

• nuttx/boards/arm64/qemu/qemu-a53

PinePhone is based on Allwinner A64 SoC with 4 Cores of Arm Cortex-A53...

• PinePhone Wiki

Will NuttX run on PinePhone? Let's find out!

Why NuttX?

NuttX is tiny and might be a fun way to teach more people about the internals of Phone Operating Systems. (Without digging deep into the entire Linux Stack)

Someday we might have a cheap, fast, responsive and tweakable phone running on NuttX!

Many thanks to qinwei2004 and the NuttX Team for implementing Cortex-A53 support!

Download NuttX

Download the Source Code for NuttX Mainline, which supports Arm Cortex-A53...

## Create NuttX Directory
mkdir nuttx
cd nuttx

## Download NuttX OS
git clone \
    --recursive \
    --branch arm64 \
    https://github.com/lupyuen/incubator-nuttx \
    nuttx

## Download NuttX Apps
git clone \
    --recursive \
    --branch arm64 \
    https://github.com/lupyuen/incubator-nuttx-apps \
    apps

## We'll build NuttX inside nuttx/nuttx
cd nuttx

Install the Build Prerequisites, skip the RISC-V Toolchain...

• "Install Prerequisites"

Download Toolchain

Download the Arm Toolchain for AArch64 ELF Bare-Metal Target (aarch64-none-elf)...

• Arm GNU Toolchain Downloads

For Linux x64 and WSL:

• gcc-arm-11.2-2022.02-x86_64-aarch64-none-elf.tar.xz

For macOS:

• arm-gnu-toolchain-11.3.rel1-darwin-x86_64-aarch64-none-elf.pkg

(I don't recommend building NuttX on Plain Old Windows CMD, please use WSL instead)

Add it to the PATH...

## For Linux x64 and WSL:
export PATH="$PATH:$HOME/gcc-arm-11.2-2022.02-x86_64-aarch64-none-elf/bin"

## For macOS:
export PATH="$PATH:/Applications/ArmGNUToolchain/11.3.rel1/aarch64-none-elf/bin"

Check the toolchain...

aarch64-none-elf-gcc -v

(Based on the instructions here)

Download QEMU

Download and install QEMU...

• Download QEMU

For macOS we may use brew...

brew install qemu

Build NuttX: Single Core

First we build NuttX for a Single Core of Arm Cortex-A53...

## Configure NuttX for Single Core
./tools/configure.sh -l qemu-a53:nsh

## Build NuttX
make

## Dump the disassembly to nuttx.S
aarch64-none-elf-objdump \
  -t -S --demangle --line-numbers --wide \
  nuttx \
  >nuttx.S \
  2>&1

The NuttX Output Files may be found here...

• NuttX for Arm Cortex-A53 Single Core

Test NuttX with QEMU: Single Core

This is how we test NuttX on QEMU with a Single Core of Arm Cortex-A53...

## Start QEMU (Single Core) with NuttX
qemu-system-aarch64 \
  -cpu cortex-a53 \
  -nographic \
  -machine virt,virtualization=on,gic-version=3 \
  -net none \
  -chardev stdio,id=con,mux=on \
  -serial chardev:con \
  -mon chardev=con,mode=readline \
  -kernel ./nuttx

Here's NuttX with a Single Core running on QEMU...

- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

nx_start: Entry
up_allocate_heap: heap_start=0x0x402c4000, heap_size=0x7d3c000
gic_validate_dist_version: GICv3 version detect
gic_validate_dist_version: GICD_TYPER = 0x37a0007
gic_validate_dist_version: 224 SPIs implemented
gic_validate_dist_version: 0 Extended SPIs implemented
gic_validate_dist_version: Distributor has no Range Selector support
gic_validate_redist_version: GICD_TYPER = 0x1000011
gic_validate_redist_version: 16 PPIs implemented
gic_validate_redist_version: no VLPI support, no direct LPI support
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 62.50MHz, cycle 62500
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x402a7000 _einit: 0x402a7000 _stext: 0x40280000 _etext: 0x402a8000
nsh: sysinit: fopen failed: 2
nsh: mkfatfs: command not found

NuttShell (NSH) NuttX-10.3.0-RC2
nsh> nx_start: CPU0: Beginning Idle Loop

nsh> help
help usage:  help [-v] [<cmd>]

  .         cd        dmesg     help      mount     rmdir     true      xd        
  [         cp        echo      hexdump   mv        set       truncate  
  ?         cmp       exec      kill      printf    sleep     uname     
  basename  dirname   exit      ls        ps        source    umount    
  break     dd        false     mkdir     pwd       test      unset     
  cat       df        free      mkrd      rm        time      usleep    

Builtin Apps:
  getprime  hello     nsh       ostest    sh        

nsh> uname -a
NuttX 10.3.0-RC2 1e8f2a8 Aug 23 2022 07:04:54 arm64 qemu-a53

nsh> hello
task_spawn: name=hello entry=0x4029b594 file_actions=0x402c9580 attr=0x402c9588 argv=0x402c96d0
spawn_execattrs: Setting policy=2 priority=100 for pid=3
Hello, World!!

nsh> ls /
/:
 dev/
 etc/
 proc/

nsh> ls /dev
/dev:
 console
 null
 ram0
 ram2
 ttyS0
 zero

nsh> ls /proc
/proc:
 0/
 1/
 2/
 meminfo
 memdump
 fs/
 self/
 uptime
 version

nsh> ls /etc
/etc:
 init.d/

nsh> ls /etc/init.d
/etc/init.d:
 rcS

nsh> cat /etc/init.d/rcS
# Create a RAMDISK and mount it at /tmp

mkrd -m 2 -s 512 1024
mkfatfs /dev/ram2
mount -t vfat /dev/ram2 /tmp

NuttX is POSIX Compliant, so the developer experience feels very much like Linux. (But much smaller)

And NuttX runs everything in RAM, no File System needed. (For now)

Build NuttX: Multi Core

From Single Core to Multi Core! Now we build NuttX for 4 Cores of Arm Cortex-A53...

## Erase the NuttX Configuration
make distclean

## Configure NuttX for 4 Cores
./tools/configure.sh -l qemu-a53:nsh_smp

## Build NuttX
make

## Dump the disassembly to nuttx.S
aarch64-none-elf-objdump \
  -t -S --demangle --line-numbers --wide \
  nuttx \
  >nuttx.S \
  2>&1

The NuttX Output Files may be found here...

• NuttX for Arm Cortex-A53 Multi-Core

Test NuttX with QEMU: Multi Core

And this is how we test NuttX on QEMU with 4 Cores of Arm Cortex-A53...

## Start QEMU (4 Cores) with NuttX
qemu-system-aarch64 \
  -smp 4 \
  -cpu cortex-a53 \
  -nographic \
  -machine virt,virtualization=on,gic-version=3 \
  -net none \
  -chardev stdio,id=con,mux=on \
  -serial chardev:con \
  -mon chardev=con,mode=readline \
  -kernel ./nuttx

Note that smp is set to 4. (Symmetric Multi-Processing)

Here's NuttX with 4 Cores running on QEMU...

- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

[CPU0] psci_detect: Detected PSCI v1.1
[CPU0] nx_start: Entry
[CPU0] up_allocate_heap: heap_start=0x0x402db000, heap_size=0x7d25000
[CPU0] gic_validate_dist_version: GICv3 version detect
[CPU0] gic_validate_dist_version: GICD_TYPER = 0x37a0007
[CPU0] gic_validate_dist_version: 224 SPIs implemented
[CPU0] gic_validate_dist_version: 0 Extended SPIs implemented
[CPU0] gic_validate_dist_version: Distributor has no Range Selector support
[CPU0] gic_validate_redist_version: GICD_TYPER = 0x1000001
[CPU0] gic_validate_redist_version: 16 PPIs implemented
[CPU0] gic_validate_redist_version: no VLPI support, no direct LPI support
[CPU0] up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 62.50MHz, cycle 62500
[CPU0] uart_register: Registering /dev/console
[CPU0] uart_register: Registering /dev/ttyS0

- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

[CPU1] gic_validate_redist_version: GICD_TYPER = 0x101000101
[CPU1] gic_validate_redist_version: 16 PPIs implemented
[CPU1] gic_validate_redist_version: no VLPI support, no direct LPI support
[CPU1] nx_idle_trampoline: CPU1: Beginning Idle Loop
[CPU0] arm64_start_cpu: Secondary CPU core 1 (MPID:0x1) is up

- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

[CPU2] gic_validate_redist_version: GICD_TYPER = 0x201000201
[CPU2] gic_validate_redist_version: 16 PPIs implemented
[CPU2] gic_validate_redist_version: no VLPI support, no direct LPI support
[CPU2] nx_idle_trampoline: CPU2: Beginning Idle Loop
[CPU0] arm64_start_cpu: Secondary CPU core 2 (MPID:0x2) is up

- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

[CPU3] gic_validate_redist_version: GICD_TYPER = 0x301000311
[CPU3] gic_validate_redist_version: 16 PPIs implemented
[CPU3] gic_validate_redist_version: no VLPI support, no direct LPI support
[CPU0] arm64_start_cpu: Secondary CPU core 3 (MPID:0x3) is up
[CPU0] work_start_highpri: Starting high-priority kernel worker thread(s)
[CPU0] nx_start_application: Starting init thread
[CPU3] nx_idle_trampoline: CPU3: Beginning Idle Loop
[CPU0] nx_start: CPU0: Beginning Idle Loop

nsh: sysinit: fopen failed: 2
nsh: mkfatfs: command not found

NuttShell (NSH) NuttX-10.3.0-RC2
nsh> help
help usage:  help [-v] [<cmd>]

  .         cd        dmesg     help      mount     rmdir     true      xd        
  [         cp        echo      hexdump   mv        set       truncate  
  ?         cmp       exec      kill      printf    sleep     uname     
  basename  dirname   exit      ls        ps        source    umount    
  break     dd        false     mkdir     pwd       test      unset     
  cat       df        free      mkrd      rm        time      usleep    

Builtin Apps:
  getprime  hello     nsh       ostest    sh        smp       taskset   

nsh> uname -a
NuttX 10.3.0-RC2 1e8f2a8 Aug 21 2022 15:57:35 arm64 qemu-a53

nsh> hello
[CPU0] task_spawn: name=hello entry=0x4029cee4 file_actions=0x402e52b0 attr=0x402e52b8 argv=0x402e5400
[CPU0] spawn_execattrs: Setting policy=2 priority=100 for pid=6
Hello, World!

We see each of the 4 Cores starting NuttX (CPU0 to CPU3). That's so cool!

(Can we use QEMU to partially emulate PinePhone? That would be extremely helpful!)

Inside NuttX for Cortex-A53

Now we browse the Source Files for the implementation of Cortex-A53 on NuttX.

NuttX treats QEMU as a Target Board (as though it was a dev board). Here are the Source Files and Build Configuration for the QEMU Board...

• nuttx/boards/arm64/qemu/qemu-a53

(We'll clone this to create a Target Board for PinePhone)

The Board-Specific Drivers for QEMU are started in qemu-a53/src/qemu_bringup.c

(We'll start the PinePhone Drivers here)

The QEMU Board calls the QEMU Architecture-Specific Drivers at...

• nuttx/arch/arm64/src/qemu

The UART Driver is located at qemu/qemu_serial.c and qemu/qemu_lowputc.S

(For PinePhone we'll create a UART Driver for Allwinner A64 SoC. I2C, SPI and other Low-Level A64 Drivers will be located here too)

The QEMU Functions (Board and Architecture) call the Arm64 Architecture Functions at...

• nuttx/arch/arm64/src/common

Which implements all kinds of Arm64 Features: FPU, Interrupts, MMU, Tasks, Timers...

(We'll reuse them for PinePhone)

NuttX Image

Next we analyse the NuttX Image with Ghidra, to understand the NuttX Image Header and Startup Code.

Here's the NuttX ELF Image nuttx analysed by Ghidra...

Note that the NuttX Image jumps to real_start (to skip the Image Header)...

40280000 4d 5a 00 91     add        x13,x18,#0x16
40280004 0f 00 00 14     b          real_start

real_start is defined at 0x4028 0040 with the Startup Code...

We see something interesting: The Magic Number ARM\x64 appears at address 0x4028 0038.

Searching the net for this Magic Number reveals that it's actually an Arm64 Linux Kernel Header!

When we refer to the NuttX Arm64 Disassembly nuttx.S, we find happiness: arch/arm64/src/common/arm64_head.S

    /* Kernel startup entry point.
     * ---------------------------
     *
     * The requirements are:
     *   MMU = off, D-cache = off, I-cache = on or off,
     *   x0 = physical address to the FDT blob.
     *       it will be used when NuttX support device tree in the future
     *
     * This must be the very first address in the loaded image.
     * It should be loaded at any 4K-aligned address.
     */
    .globl __start;
__start:

    /* DO NOT MODIFY. Image header expected by Linux boot-loaders.
     *
     * This add instruction has no meaningful effect except that
     * its opcode forms the magic "MZ" signature of a PE/COFF file
     * that is required for UEFI applications.
     *
     * Some bootloader (such imx8 uboot) checking the magic "MZ" to see
     * if the image is a valid Linux image. but modifying the bootLoader is
     * unnecessary unless we need to do a customize secure boot.
     * so just put the ''MZ" in the header to make bootloader happiness
     */

    add     x13, x18, #0x16      /* the magic "MZ" signature */
    b       real_start           /* branch to kernel start */
    .quad   0x480000              /* Image load offset from start of RAM */
    .quad   _e_initstack - __start         /* Effective size of kernel image, little-endian */
    .quad   __HEAD_FLAGS         /* Informative flags, little-endian */
    .quad   0                    /* reserved */
    .quad   0                    /* reserved */
    .quad   0                    /* reserved */
    .ascii  "ARM\x64"            /* Magic number, "ARM\x64" */
    .long   0                    /* reserved */

real_start:
    /* Disable all exceptions and interrupts */

NuttX Image actually follows the Arm64 Linux Kernel Image Format! As defined here...

• "Booting AArch64 Linux"

Arm64 Linux Kernel Image contains a 64-byte header...

u32 code0;                    /* Executable code */
u32 code1;                    /* Executable code */
u64 text_offset;              /* Image load offset, little endian */
u64 image_size;               /* Effective Image size, little endian */
u64 flags;                    /* kernel flags, little endian */
u64 res2      = 0;            /* reserved */
u64 res3      = 0;            /* reserved */
u64 res4      = 0;            /* reserved */
u32 magic     = 0x644d5241;   /* Magic number, little endian, "ARM\x64" */
u32 res5;                     /* reserved (used for PE COFF offset) */

Start of RAM is 0x4000 0000. The Image Load Offset in our NuttX Image Header is 0x48 0000 according to arch/arm64/src/common/arm64_head.S

    .quad   0x480000              /* Image load offset from start of RAM */

This means that our NuttX Image will be loaded at 0x4048 0000.

I wonder if this Image Load Offset should have been 0x28 0000? (Instead of 0x48 0000)

Remember that Ghidra (and the Arm Disassembly) says that our NuttX Image is actually loaded at 0x4028 0000. (Instead of 0x4048 0000)

RAM Size and RAM Start are defined in the NuttX Configuration: boards/arm64/qemu/qemu-a53/configs/nsh_smp/defconfig

CONFIG_RAM_SIZE=134217728
CONFIG_RAM_START=0x40000000

That's 128 MB RAM. Which should fit inside PinePhone's 2 GB RAM.

The NuttX Image was built with this Linker Command, based on make --trace...

aarch64-none-elf-ld \
  --entry=__start \
  -nostdlib \
  --cref \
  -Map=nuttx/nuttx/nuttx.map \
  -Tnuttx/nuttx/boards/arm64/qemu/qemu-a53/scripts/dramboot.ld  \
  -L nuttx/nuttx/staging \
  -L nuttx/nuttx/arch/arm64/src/board  \
  -o nuttx/nuttx/nuttx arm64_head.o  \
  --start-group \
  -lsched \
  -ldrivers \
  -lboards \
  -lc \
  -lmm \
  -larch \
  -lapps \
  -lfs \
  -lbinfmt \
  -lboard /Applications/ArmGNUToolchain/11.3.rel1/aarch64-none-elf/bin/../lib/gcc/aarch64-none-elf/11.3.1/libgcc.a /Applications/ArmGNUToolchain/11.3.rel1/aarch64-none-elf/bin/../lib/gcc/aarch64-none-elf/11.3.1/../../../../aarch64-none-elf/lib/libm.a \
  --end-group

NuttX Image begins at __start, which is defined as 0x4028 0000 in the NuttX Linker Script: boards/arm64/qemu/qemu-a53/scripts/dramboot.ld

SECTIONS
{
  . = 0x40280000;  /* uboot load address */
  _start = .;

We'll change this to 0x4008 0000 for PinePhone, since Kernel Start Address is 0x4008 0000 and Image Load Offset is 0. (See below)

We've seen the NuttX Image (which looks like a Linux Kernel Image), let's compare with a PinePhone Linux Kernel Image and see how NuttX needs to be tweaked...

PinePhone Image

Will NuttX run on PinePhone? Let's analyse a PinePhone Linux Kernel Image with Ghidra, to look at the Linux Kernel Header and Startup Code.

We'll use the PinePhone Jumpdrive Image, since it's small...

https://github.com/dreemurrs-embedded/Jumpdrive

Download https://github.com/dreemurrs-embedded/Jumpdrive/releases/download/0.8/pine64-pinephone.img.xz

Expand pine64-pinephone.img.xz

Expand the files inside...

gunzip Image.gz
gunzip initramfs.gz
tar xvf initramfs

Import the uncompressed Image (Linux Kernel) into Ghidra.

For "Language" select AARCH64:LE:v8A:default...

• Processor: AARCH64
• Variant: v8A
• Size: 64
• Endian: little
• Compiler: default

Here's the Jumpdrive Image (Linux Kernel) in Ghidra...

According to the Linux Kernel Header...

• "Booting AArch64 Linux"

We see Linux Kernel Magic Number ARM\x64 at offset 0x38.

Image Load Offset is 0, according to the header.

Kernel Start Address on PinePhone is 0x4008 0000.

So we shift Image in Ghidra to start at 0x4008 0000...

• Click Window > Memory Map

• Click "ram"

• Click the 4-Arrows icon ("Move a block to another address")

• Change "New Start Address" to 40080000

Will NuttX Boot On PinePhone?

So will NuttX boot on PinePhone?

It's highly plausible! We discovered (with happiness) that NuttX already generates an Arm64 Linux Kernel Header.

So NuttX could be a drop-in replacement for the PinePhone Linux Kernel! We just need to...

• Write PinePhone Jumpdrive to a microSD Card (with Etcher, in FAT format)

• Overwrite Image.gz by the (gzipped) NuttX Binary Image nuttx.bin.gz

• Insert the microSD Card into PinePhone

• Power on PinePhone

And NuttX should (theoretically) boot on PinePhone!

As mentioned earlier, we should rebuild NuttX so that __start is changed to 0x4008 0000 (from 0x4028 0000), as defined in the NuttX Linker Script: boards/arm64/qemu/qemu-a53/scripts/dramboot.ld

SECTIONS
{
SECTIONS
{
  . = 0x40080000;  /* PinePhone uboot load address (kernel_addr_r) */
  /* Previously: . = 0x40280000; */  /* uboot load address */
  _start = .;

Also the Image Load Offset in our NuttX Image Header should be changed to 0x0 (from 0x48 0000): arch/arm64/src/common/arm64_head.S

    .quad   0x0000               /* PinePhone Image load offset from start of RAM */
    # Previously: .quad   0x480000              /* Image load offset from start of RAM */

Later we'll increase the RAM Size to 2 GB (from 128 MB): boards/arm64/qemu/qemu-a53/configs/nsh_smp/defconfig

/* TODO: Increase to 2 GB for PinePhone */
CONFIG_RAM_SIZE=134217728
CONFIG_RAM_START=0x40000000

But not right now, because it might clash with the Device Tree and RAM File System.

But will we see anything when NuttX boots on PinePhone?

Not yet. We'll need to implement the UART Driver for NuttX...

UART Driver for NuttX

We won't see any output from NuttX until we implement the UART Driver for NuttX.

These are the Source Files for the QEMU UART Driver (PL011)...

• arch/arm64/src/qemu/qemu_serial.c

• arch/arm64/src/qemu/qemu_lowputc.S

  (More about PL011 UART)

We'll replace the code above with the UART Driver for Allwinner A64 SoC...

• UART0 Memory Map

• Allwinner A64 UART

• Allwinner A64 User Manual

• Allwinner A64 Info

To access the UART Port on PinePhone, we'll use this USB Serial Debug Cable...

• PinePhone Serial Debug Cable

Which connects to the Headphone Port. Genius!

(Remember to flip the Headphone Switch to OFF)

PinePhone UART Port in disguise

PinePhone U-Boot Log

Before starting the Linux Kernel, PinePhone boots by running the U-Boot Bootloader...

• A64 Boot ROM

• A64 U-Boot

• A64 U-Boot SPL

• SD Card Layout

Here's the PinePhone U-Boot Log captured with the USB Serial Debug Cable...

(Press Enter repeatedly when PinePhone powers on to enter the U-Boot Prompt)

$ screen /dev/ttyUSB0 115200

DRAM: 2048 MiB
Trying to boot from MMC1
NOTICE:  BL31: v2.2(release):v2.2-904-gf9ea3a629
NOTICE:  BL31: Built : 15:32:12, Apr  9 2020
NOTICE:  BL31: Detected Allwinner A64/H64/R18 SoC (1689)
NOTICE:  BL31: Found U-Boot DTB at 0x4064410, model: PinePhone
NOTICE:  PSCI: System suspend is unavailable

U-Boot 2020.07 (Nov 08 2020 - 00:15:12 +0100)

DRAM:  2 GiB
MMC:   Device 'mmc@1c11000': seq 1 is in use by 'mmc@1c10000'
mmc@1c0f000: 0, mmc@1c10000: 2, mmc@1c11000: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment

starting USB...
No working controllers found
Hit any key to stop autoboot:

=> help
?         - alias for 'help'
base      - print or set address offset
bdinfo    - print Board Info structure
blkcache  - block cache diagnostics and control
boot      - boot default, i.e., run 'bootcmd'
bootd     - boot default, i.e., run 'bootcmd'
bootelf   - Boot from an ELF image in memory
booti     - boot Linux kernel 'Image' format from memory
bootm     - boot application image from memory
bootvx    - Boot vxWorks from an ELF image
cmp       - memory compare
coninfo   - print console devices and information
cp        - memory copy
crc32     - checksum calculation
dm        - Driver model low level access
echo      - echo args to console
editenv   - edit environment variable
env       - environment handling commands
exit      - exit script
ext2load  - load binary file from a Ext2 filesystem
ext2ls    - list files in a directory (default /)
ext4load  - load binary file from a Ext4 filesystem
ext4ls    - list files in a directory (default /)
ext4size  - determine a file's size
false     - do nothing, unsuccessfully
fatinfo   - print information about filesystem
fatload   - load binary file from a dos filesystem
fatls     - list files in a directory (default /)
fatmkdir  - create a directory
fatrm     - delete a file
fatsize   - determine a file's size
fatwrite  - write file into a dos filesystem
fdt       - flattened device tree utility commands
fstype    - Look up a filesystem type
go        - start application at address 'addr'
gpio      - query and control gpio pins
gpt       - GUID Partition Table
gzwrite   - unzip and write memory to block device
help      - print command description/usage
iminfo    - print header information for application image
imxtract  - extract a part of a multi-image
itest     - return true/false on integer compare
ln        - Create a symbolic link
load      - load binary file from a filesystem
loadb     - load binary file over serial line (kermit mode)
loads     - load S-Record file over serial line
loadx     - load binary file over serial line (xmodem mode)
loady     - load binary file over serial line (ymodem mode)
loop      - infinite loop on address range
ls        - list files in a directory (default /)
lzmadec   - lzma uncompress a memory region
md        - memory display
mm        - memory modify (auto-incrementing address)
mmc       - MMC sub system
mmcinfo   - display MMC info
mw        - memory write (fill)
nm        - memory modify (constant address)
part      - disk partition related commands
poweroff  - Perform POWEROFF of the device
printenv  - print environment variables
random    - fill memory with random pattern
reset     - Perform RESET of the CPU
run       - run commands in an environment variable
save      - save file to a filesystem
saveenv   - save environment variables to persistent storage
setenv    - set environment variables
setexpr   - set environment variable as the result of eval expression
sf        - SPI flash sub-system
showvar   - print local hushshell variables
size      - determine a file's size
sleep     - delay execution for some time
source    - run script from memory
sysboot   - command to get and boot from syslinux files
test      - minimal test like /bin/sh
true      - do nothing, successfully
unlz4     - lz4 uncompress a memory region
unzip     - unzip a memory region
usb       - USB sub-system
usbboot   - boot from USB device
version   - print monitor, compiler and linker version

=> printenv
arch=arm
baudrate=115200
board=sunxi
board_name=sunxi
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_net_usb_start=usb start
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=fel mmc_auto usb0 
bootcmd=run distro_bootcmd
bootcmd_fel=if test -n ${fel_booted} && test -n ${fel_scriptaddr}; then echo '(FEL boot)'; source ${fel_scriptaddr}; fi
bootcmd_mmc0=devnum=0; run mmc_boot
bootcmd_mmc1=devnum=1; run mmc_boot
bootcmd_mmc_auto=if test ${mmc_bootdev} -eq 1; then run bootcmd_mmc1; run bootcmd_mmc0; elif test ${mmc_bootdev} -eq 0; then run bootcmd_mmc0; run bootcmd_mmc1; fi
bootcmd_usb0=devnum=0; run usb_boot
bootdelay=0
bootm_size=0xa000000
console=ttyS0,115200
cpu=armv8
dfu_alt_info_ram=kernel ram 0x40080000 0x1000000;fdt ram 0x4FA00000 0x100000;ramdisk ram 0x4FE00000 0x4000000
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
ethaddr=02:ba:8c:73:bf:ca
fdt_addr_r=0x4FA00000
fdtcontroladdr=bbf4dd40
fdtfile=allwinner/sun50i-a64-pinephone.dtb
kernel_addr_r=0x40080000
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
mmc_bootdev=0
partitions=name=loader1,start=8k,size=32k,uuid=${uuid_gpt_loader1};name=loader2,size=984k,uuid=${uuid_gpt_loader2};name=esp,size=128M,bootable,uuid=${uuid_gpt_esp};name=system,size=-,uuid=${uuid_gpt_system};
preboot=usb start
pxefile_addr_r=0x4FD00000
ramdisk_addr_r=0x4FE00000
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scriptaddr=0x4FC00000
serial#=92c07dba8c73bfca
soc=sunxi
stderr=serial@1c28000
stdin=serial@1c28000
stdout=serial@1c28000
usb_boot=usb start; if usb dev ${devnum}; then devtype=usb; run scan_dev_for_boot_part; fi
uuid_gpt_esp=c12a7328-f81f-11d2-ba4b-00a0c93ec93b
uuid_gpt_system=b921b045-1df0-41c3-af44-4c6f280d3fae

Environment size: 2861/131068 bytes

=> boot
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
653 bytes read in 3 ms (211.9 KiB/s)
## Executing script at 4fc00000
gpio: pin 114 (gpio 114) value is 1
4275261 bytes read in 192 ms (21.2 MiB/s)
Uncompressed size: 10170376 = 0x9B3008
36162 bytes read in 4 ms (8.6 MiB/s)
1078500 bytes read in 51 ms (20.2 MiB/s)
## Flattened Device Tree blob at 4fa00000
   Booting using the fdt blob at 0x4fa00000
   Loading Ramdisk to 49ef8000, end 49fff4e4 ... OK
   Loading Device Tree to 0000000049eec000, end 0000000049ef7d41 ... OK

Starting kernel ...

/ # 

According to the U-Boot Log, the Start of RAM kernel_addr_r is 0x4008 0000.

We need to set this in the NuttX Linker Script and the NuttX Header...

NuttX Boots On PinePhone

In the previous section, U-Boot says that the Start of RAM kernel_addr_r is 0x4008 0000.

Let's set this in the NuttX Linker Script and the NuttX Header...

• Change Image Load Offset in NuttX Header to 0x0 (from 0x48000)

  (See the changes)

• Change NuttX Linker Script to set the Start Address _start to 0x4008 0000 (from 0x4028 0000)

  (See the changes)

For PinePhone Allwinner A64 UART: We reused the previous code for transmitting output to UART...

/* PL011 UART transmit character
 * xb: register which contains the UART base address
 * wt: register which contains the character to transmit
 */

.macro early_uart_transmit xb, wt
    strb  \wt, [\xb]             /* -> UARTDR (Data Register) */
.endm

(Source)

But we updated the UART Register Address for Allwinner A64 UART...

 /* 32-bit register definition for qemu pl011 uart */

 /* PinePhone Allwinner A64 UART0 Base Address: */
 #define UART1_BASE_ADDRESS 0x01C28000
 /* Previously: #define UART1_BASE_ADDRESS 0x9000000 */
 #define EARLY_UART_PL011_BAUD_RATE  115200

(Source)

Right now we don't check if UART is ready to transmit, so our UART output will have missing characters. This needs to be fixed...

/* PL011 UART wait UART to be ready to transmit
 * xb: register which contains the UART base address
 * c: scratch register number
 */

.macro early_uart_ready xb, wt
1:
    # TODO: Wait for PinePhone Allwinner A64 UART
    # ldrh  \wt, [\xb, #0x18]      /* <- UARTFR (Flag register) */
    # tst   \wt, #0x8              /* Check BUSY bit */
    # b.ne  1b                     /* Wait for the UART to be ready */
.endm

(Source)

We don't init the UART Port because U-Boot has kindly done it for us. This needs to be fixed...

/* PL011 UART initialization
 * xb: register which contains the UART base address
 * c: scratch register number
 */

GTEXT(up_earlyserialinit)
SECTION_FUNC(text, up_earlyserialinit)
    # TODO: Set PinePhone Allwinner A64 Baud Rate Divisor: UART_LCR (DLAB), UART_DLL, UART_DLH
    # ldr   x15, =UART1_BASE_ADDRESS
    # mov   x0, #(7372800 / EARLY_UART_PL011_BAUD_RATE % 16)
    # strh  w0, [x15, #0x28]      /* -> UARTFBRD (Baud divisor fraction) */
    # mov   x0, #(7372800 / EARLY_UART_PL011_BAUD_RATE / 16)
    # strh  w0, [x15, #0x24]      /* -> UARTIBRD (Baud divisor integer) */
    # mov   x0, #0x60             /* 8n1 */
    # str   w0, [x15, #0x2C]      /* -> UARTLCR_H (Line control) */
    # ldr   x0, =0x00000301       /* RXE | TXE | UARTEN */
    # str   w0, [x15, #0x30]      /* -> UARTCR (Control Register) */
    ret

(Source)

With the above changes, NuttX boots on PinePhone yay!

NuttX Boot Log

This is how we build NuttX for PinePhone...

## Download NuttX OS for PinePhone
git clone \
    --recursive \
    --branch pinephone \
    https://github.com/lupyuen/incubator-nuttx \
    nuttx

## Download NuttX Apps for PinePhone
git clone \
    --recursive \
    --branch pinephone \
    https://github.com/lupyuen/incubator-nuttx-apps \
    apps

## We'll build NuttX inside nuttx/nuttx
cd nuttx

## Configure NuttX for Single Core
./tools/configure.sh -l qemu-a53:nsh

## Build NuttX
make

## Dump the disassembly to nuttx.S
aarch64-none-elf-objdump \
  -t -S --demangle --line-numbers --wide \
  nuttx \
  >nuttx.S \
  2>&1

## Compress the NuttX Binary Image
cp nuttx.bin Image
rm -f Image.gz
gzip Image

## Copy compressed NuttX Binary Image to Jumpdrive microSD
## https://lupyuen.github.io/articles/uboot#pinephone-jumpdrive
## TODO: Change the microSD Path
cp Image.gz "/Volumes/NO NAME"

Insert the Jumpdrive microSD into PinePhone and power up.

Here's the UART Log of NuttX booting on PinePhone...

DRAM: 2048 MiB
Trying to boot from MMC1
NOTICE:  BL31: v2.2(release):v2.2-904-gf9ea3a629
NOTICE:  BL31: Built : 15:32:12, Apr  9 2020
NOTICE:  BL31: Detected Allwinner A64/H64/R18 SoC (1689)
NOTICE:  BL31: Found U-Boot DTB at 0x4064410, model: PinePhone
NOTICE:  PSCI: System suspend is unavailable

U-Boot 2020.07 (Nov 08 2020 - 00:15:12 +0100)

DRAM:  2 GiB
MMC:   Device 'mmc@1c11000': seq 1 is in use by 'mmc@1c10000'
mmc@1c0f000: 0, mmc@1c10000: 2, mmc@1c11000: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment

starting USB...
No working controllers found
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
653 bytes read in 3 ms (211.9 KiB/s)
## Executing script at 4fc00000
gpio: pin 114 (gpio 114) value is 1
99784 bytes read in 8 ms (11.9 MiB/s)
Uncompressed size: 278528 = 0x44000
36162 bytes read in 4 ms (8.6 MiB/s)
1078500 bytes read in 51 ms (20.2 MiB/s)
## Flattened Device Tree blob at 4fa00000
   Booting using the fdt blob at 0x4fa00000
   Loading Ramdisk to 49ef8000, end 49fff4e4 ... OK
   Loading Device Tree to 0000000049eec000, end 0000000049ef7d41 ... OK

Starting kernel ...

HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x400c4000, heap_size=0x7f3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
up_timer_initialize: _vector_table=0x400a7000
up_timer_initialize: Before writing: vbar_el1=0x40227000
up_timer_initialize: After writing: vbar_el1=0x400a7000
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x400a7000 _einit: 0x400a7000 _stext: 0x40080000 _etext: 0x400a8000
nx_start: CPU0: Beginning Idle Loop

Where's the rest of the boot output?

We expect to see this output when NuttX boots...

• "Test NuttX: Single Core"

But PinePhone stops halfway. Let's find out why...

Interrupt Controller

Let's talk about the Arm Generic Interrupt Controller (GIC) for PinePhone...

arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2

This is the current implementation of Arm GIC Version 3 in NuttX Arm64...

• arch/arm64/src/common/arm64_gicv3.c

• arch/arm64/src/common/arm64_gic.h

This implementation won't work on PinePhone, so we have commented out the existing code and inserted our own implementation.

Why won't Arm GIC Version 3 work on PinePhone?

According to the Allwinner A64 SoC User Manual (page 210, "GIC"), PinePhone's Interrupt Controller runs on...

• Arm GIC PL400, which is based on...

• Arm GIC Version 2

We'll have to downgrade arm64_gicv3.c to support Arm GIC Version 2 for PinePhone.

Does NuttX implement Arm GIC Version 2?

NuttX has an implementation of Arm GIC Version 2, but it's based on Arm32. We'll port it from Arm32 to Arm64...

• arch/arm/src/armv7-a/arm_gicv2.c

• arch/arm/src/armv7-a/arm_gicv2_dump.c

• arch/arm/src/armv7-a/gic.h

• arch/arm/src/armv7-a/mpcore.h

• arch/arm/src/imx6/chip.h

• arch/arm/src/imx6/hardware/imx_memorymap.h

By reusing the code above, we have implemented Arm GIC Version 2 for PinePhone...

• arch/arm64/src/common/arm64_gicv3.c

We made minor tweaks to NuttX's implementation of GIC Version 2...

• Changes for arch/arm/src/armv7-a/arm_gicv2.c

• Changes for arch/arm/src/armv7-a/arm_gicv2_dump.c

• Changes for arch/arm/src/armv7-a/gic.h

Where in memory is the GIC located?

According to the Allwinner A64 SoC User Manual (page 74, "Memory Mapping"), the GIC is located at this address...

Module	Address (It is for Cluster CPU)	Remarks
SCU space	0x01C80000	(What's this?)
	GIC_DIST: 0x01C80000 + 0x1000	GIC Distributor (GICD)
CPUS can’t access	GIC_CPUIF:0x01C80000 + 0x2000	GIC CPU Interface (GICC)

(Why "CPUS can’t access"?)

The Interrupt Sources are defined in the Allwinner A64 SoC User Manual (page 210, "GIC")...

• 16 x Software-Generated Interrupts (SGI)

  "This is an interrupt generated by software writing to a GICD_SGIR register in the GIC. The system uses SGIs for interprocessor communication."

• 16 x Private Peripheral Interrupts (PPI)

  "This is a peripheral interrupt that is specific to a single processor"

• 125 x Shared Peripheral Interrupts (SPI)

  "This is a peripheral interrupt that the Distributor can route to any of a specified combination of processors"

To verify the GIC Version, read the Peripheral ID2 Register (ICPIDR2) at Offset 0xFE8 of GIC Distributor.

Bits 4 to 7 of ICPIDR2 are...

• 0x1 for GIC Version 1
• 0x2 for GIC Version 2

This is how we implement the GIC Version verification: arch/arm64/src/common/arm64_gicv3.c

// Init GIC v2 for PinePhone. See https://github.com/lupyuen/pinephone-nuttx#interrupt-controller
int arm64_gic_initialize(void)
{
  sinfo("TODO: Init GIC for PinePhone\n");

  // To verify the GIC Version, read the Peripheral ID2 Register (ICPIDR2) at Offset 0xFE8 of GIC Distributor.
  // Bits 4 to 7 of ICPIDR2 are...
  // - 0x1 for GIC Version 1
  // - 0x2 for GIC Version 2
  // GIC Distributor is at 0x01C80000 + 0x1000.
  // See https://github.com/lupyuen/pinephone-nuttx#interrupt-controller
  const uint8_t *ICPIDR2 = (const uint8_t *) (CONFIG_GICD_BASE + 0xFE8);
  uint8_t version = (*ICPIDR2 >> 4) & 0b1111;
  sinfo("GIC Version is %d\n", version);
  DEBUGASSERT(version == 2);

  // arm_gic0_initialize must be called on CPU0
  arm_gic0_initialize();

  // arm_gic_initialize must be called for all CPUs
  // TODO: Move to arm64_gic_secondary_init
  arm_gic_initialize();

  return 0;
}

See below for the GIC Register Dump.

Let's talk about NuttX's System Timer, which depends on the GIC...

System Timer

NuttX starts the System Timer when it boots. Here's how the System Timer is started: arch/arm64/src/common/arm64_arch_timer.c

void up_timer_initialize(void)
{
  uint64_t curr_cycle;

  arch_timer_rate   = arm64_arch_timer_get_cntfrq();
  cycle_per_tick    = ((uint64_t)arch_timer_rate / (uint64_t)TICK_PER_SEC);

  sinfo("%s: cp15 timer(s) running at %lu.%02luMHz, cycle %ld\n", __func__,
        (unsigned long)arch_timer_rate / 1000000,
        (unsigned long)(arch_timer_rate / 10000) % 100, cycle_per_tick);

  irq_attach(ARM_ARCH_TIMER_IRQ, arm64_arch_timer_compare_isr, 0);
  arm64_gic_irq_set_priority(ARM_ARCH_TIMER_IRQ, ARM_ARCH_TIMER_PRIO,
                             ARM_ARCH_TIMER_FLAGS);

  curr_cycle = arm64_arch_timer_count();
  arm64_arch_timer_set_compare(curr_cycle + cycle_per_tick);
  arm64_arch_timer_enable(true);

  up_enable_irq(ARM_ARCH_TIMER_IRQ);
  arm64_arch_timer_set_irq_mask(false);
}

At every tick, the System Timer triggers an interrupt that calls arm64_arch_timer_compare_isr

(CONFIG_SCHED_TICKLESS is undefined)

Timer IRQ ARM_ARCH_TIMER_IRQ is defined in arch/arm64/src/common/arm64_arch_timer.h

#define CONFIG_ARM_TIMER_SECURE_IRQ         (GIC_PPI_INT_BASE + 13)
#define CONFIG_ARM_TIMER_NON_SECURE_IRQ     (GIC_PPI_INT_BASE + 14)
#define CONFIG_ARM_TIMER_VIRTUAL_IRQ        (GIC_PPI_INT_BASE + 11)
#define CONFIG_ARM_TIMER_HYP_IRQ            (GIC_PPI_INT_BASE + 10)

#define ARM_ARCH_TIMER_IRQ	CONFIG_ARM_TIMER_VIRTUAL_IRQ
#define ARM_ARCH_TIMER_PRIO	IRQ_DEFAULT_PRIORITY
#define ARM_ARCH_TIMER_FLAGS	IRQ_TYPE_LEVEL

GIC_PPI_INT_BASE is defined in arch/arm64/src/common/arm64_gic.h

#define GIC_SGI_INT_BASE            0
#define GIC_PPI_INT_BASE            16
#define GIC_IS_SGI(intid)           (((intid) >= GIC_SGI_INT_BASE) && \
                                     ((intid) < GIC_PPI_INT_BASE))

#define GIC_SPI_INT_BASE            32
#define GIC_NUM_INTR_PER_REG        32
#define GIC_NUM_CFG_PER_REG         16
#define GIC_NUM_PRI_PER_REG         4

Timer Interrupt Isn't Handled

Previously NuttX hangs midsentence while booting on PinePhone, let's find out how we fixed it...

arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
uart_regi

Based on our experiments, it seems the System Timer triggered a Timer Interrupt, and NuttX hangs while attempting to handle the Timer Interrupt.

The Timer Interrupt Handler arm64_arch_timer_compare_isr is never called. (We checked using up_putc)

Is it caused by PinePhone's GIC?

This problem doesn't seem to be caused by PinePhone's Generic Interrupt Controller (GIC) that we have implemented. We successfully tested PinePhone's GIC with QEMU.

Let's troubleshoot the Timer Interrupt...

• We called up_putc to understand how Interrupts are handled on NuttX.

  We also added Debug Code to the Arm64 Interrupt Handler.

  (Maybe we should have used GDB with QEMU)

• We dumped the Interrupt Vector Table.

  We verified that the Timer Interrupt Handler Address in the table is correct.

• We confirmed that Interrupt Dispatcher irq_dispatch isn't called.

  And Unexpected Interrupt Handler irq_unexpected_isr isn't called either.

• Let's backtrack, maybe there's a problem in the Arm64 Interrupt Handler?

  But arm64_enter_exception and arm64_irq_handler aren't called either.

• Maybe the Arm64 Vector Table _vector_table isn't correctly configured?

  arch/arm64/src/common/arm64_vector_table.S

And we're right! The Arm64 Vector Table is indeed incorrectly configured! Here why...

Arm64 Vector Table Is Wrong

Earlier we saw that the Interrupt Handler wasn't called for System Timer Interrupt. And it might be due to problems in the Arm64 Vector Table _vector_table: arch/arm64/src/common/arm64_vector_table.S

Let's check whether the Arm64 Vector Table _vector_table is correctly configured in the Arm CPU: arch/arm64/src/common/arm64_arch_timer.c

void up_timer_initialize(void)
{
  ...
  // Attach System Timer Interrupt Handler
  irq_attach(ARM_ARCH_TIMER_IRQ, arm64_arch_timer_compare_isr, 0);

  // For PinePhone: Read Vector Base Address Register EL1
  extern void *_vector_table[];
  sinfo("_vector_table=%p\n", _vector_table);
  sinfo("Before writing: vbar_el1=%p\n", read_sysreg(vbar_el1));

After attaching the Interrupt Handler for System Timer, we read the Arm64 Vector Base Address Register EL1. Here's the output...

up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
up_timer_initialize: _vector_table=0x400a7000
up_timer_initialize: Before writing: vbar_el1=0x40227000

Aha! _vector_table is at 0x400a7000... But Vector Base Address Register EL1 says 0x40227000!

Our Arm64 CPU is pointing to the wrong Arm64 Vector Table... Hence our Interrupt Handler is never called!

Let's fix the Vector Base Address Register EL1: arch/arm64/src/common/arm64_arch_timer.c

  // For PinePhone: Write Vector Base Address Register EL1
  write_sysreg((uint64_t)_vector_table, vbar_el1);
  ARM64_ISB();

  // For PinePhone: Read Vector Base Address Register EL1
  sinfo("After writing: vbar_el1=%p\n", read_sysreg(vbar_el1));

This writes the correct value of _vector_table back into Vector Base Address Register EL1. Here's the output...

up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
up_timer_initialize: _vector_table=0x400a7000
up_timer_initialize: Before writing: vbar_el1=0x40227000
up_timer_initialize: After writing: vbar_el1=0x400a7000

Yep Vector Base Address Register EL1 is now correct.

And our Interrupt Handlers are now working fine yay!

Test PinePhone GIC with QEMU

This is how we build NuttX for QEMU with Generic Interrupt Controller (GIC) Version 2...

## Download NuttX OS for QEMU with GIC Version 2
git clone \
    --recursive \
    --branch gicv2 \
    https://github.com/lupyuen/incubator-nuttx \
    nuttx

## Download NuttX Apps for QEMU
git clone \
    --recursive \
    --branch arm64 \
    https://github.com/lupyuen/incubator-nuttx-apps \
    apps

## We'll build NuttX inside nuttx/nuttx
cd nuttx

## Configure NuttX for Single Core
./tools/configure.sh -l qemu-a53:nsh

## Build NuttX
make

## Dump the disassembly to nuttx.S
aarch64-none-elf-objdump \
  -t -S --demangle --line-numbers --wide \
  nuttx \
  >nuttx.S \
  2>&1

And this is how we tested PinePhone's GIC Version 2 with QEMU...

## Run GIC v2 with QEMU
qemu-system-aarch64 \
  -smp 4 \
  -cpu cortex-a53 \
  -nographic \
  -machine virt,virtualization=on,gic-version=2 \
  -net none \
  -chardev stdio,id=con,mux=on \
  -serial chardev:con \
  -mon chardev=con,mode=readline \
  -kernel ./nuttx

Note that gic-version=2, instead of the usual GIC Version 3 for NuttX Arm64.

Also we simulated 4 Cores of Arm Cortex-A53 (similar to PinePhone): -smp 4

QEMU boots OK with PinePhone's GIC Version 2...

- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x402c4000, heap_size=0x7d3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x8000000
arm64_gic_initialize: CONFIG_GICR_BASE=0x8010000
arm64_gic_initialize: GIC Version is 2
EFGHup_timer_initialize: up_timer_initialize: cp15 timer(s) running at 62.50MHz, cycle 62500
AKLMNOPBIJuart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
AKLMNOPBIJwork_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x402a7000 _einit: 0x402a7000 _stext: 0x40280000 _etext: 0x402a8000
nsh: sysinit: fopen failed: 2
nsh: mkfatfs: command not found

NuttShell (NSH) NuttX-10.3.0-RC2
nsh> nx_start: CPU0: Beginning Idle Loop

So our implementation of GIC Version 2 for PinePhone is probably OK.

Is the Timer Interrupt triggered correctly with PinePhone GIC?

Yes, we verified that the Timer Interrupt Handler arm64_arch_timer_compare_isr is called periodically. (We checked using up_putc)

How did we get the GIC Base Addresses?

arm64_gic_initialize: CONFIG_GICD_BASE=0x8000000
arm64_gic_initialize: CONFIG_GICR_BASE=0x8010000

We got the GIC v2 Base Addresses for GIC Distributor (CONFIG_GICD_BASE) and GIC CPU Interface (CONFIG_GICR_BASE) by dumping the Device Tree from QEMU...

## GIC v2 Dump Device Tree
qemu-system-aarch64 \
  -smp 4 \
  -cpu cortex-a53 \
  -nographic \
  -machine virt,virtualization=on,gic-version=2,dumpdtb=gicv2.dtb \
  -net none \
  -chardev stdio,id=con,mux=on \
  -serial chardev:con \
  -mon chardev=con,mode=readline \
  -kernel ./nuttx

## Convert Device Tree to text format
dtc -o gicv2.dts -O dts -I dtb gicv2.dtb

The Base Addresses are revealed in the GIC v2 Device Tree: gicv2.dts...

intc@8000000 {
reg = <
    0x00 0x8000000 0x00 0x10000  //  GIC Distributor:   0x8000000
    0x00 0x8010000 0x00 0x10000  //  GIC CPU Interface: 0x8010000
    0x00 0x8030000 0x00 0x10000  //  VGIC Virtual Interface Control: 0x8030000
    0x00 0x8040000 0x00 0x10000  //  VGIC Virtual CPU Interface:     0x8040000
>;
compatible = "arm,cortex-a15-gic";

(More about this)

We defined the Base Addresses in arch/arm64/include/qemu/chip.h

Compare the above Base Addresses with the GIC v3 Device Tree: gicv3.dts

intc@8000000 {
reg = <
    0x00 0x8000000 0x00 0x10000   //  GIC Distributor:   0x8000000
    0x00 0x80a0000 0x00 0xf60000  //  GIC CPU Interface: 0x80a0000
>;
#redistributor-regions = <0x01>;
compatible = "arm,gic-v3";

This is how we copied the PinePhone GIC v2 Source Files into NuttX Arm64 for testing...

cp ~/PinePhone/nuttx/nuttx/arch/arm64/src/common/arm64_gicv3.c      ~/gicv2/nuttx/nuttx/arch/arm64/src/common/arm64_gicv3.c
cp ~/PinePhone/nuttx/nuttx/arch/arm/src/armv7-a/arm_gicv2.c         ~/gicv2/nuttx/nuttx/arch/arm/src/armv7-a/arm_gicv2.c
cp ~/PinePhone/nuttx/nuttx/arch/arm/src/armv7-a/gic.h               ~/gicv2/nuttx/nuttx/arch/arm/src/armv7-a/gic.h
cp ~/PinePhone/nuttx/nuttx/arch/arm/src/armv7-a/arm_gicv2_dump.c    ~/gicv2/nuttx/nuttx/arch/arm/src/armv7-a/arm_gicv2_dump.c
cp ~/PinePhone/nuttx/nuttx/arch/arm64/src/common/arm64_arch_timer.c ~/gicv2/nuttx/nuttx/arch/arm64/src/common/arm64_arch_timer.c
cp ~/PinePhone/nuttx/run.sh             ~/gicv2/nuttx/run.sh
cp ~/PinePhone/nuttx/.vscode/tasks.json ~/gicv2/nuttx/.vscode/tasks.json

Handling Interrupts

Let's talk about NuttX and how it handles interrupts.

The Interrupt Vector Table is defined in sched/irq/irq_initialize.c

/* This is the interrupt vector table */
struct irq_info_s g_irqvector[NR_IRQS];

(Next section talks about dumping the Interrupt Vector Table)

At startup, the Interrupt Vector Table is initialised to the Unexpected Interrupt Handler irq_unexpected_isr: sched/irq/irq_initialize.c

/****************************************************************************
 * Name: irq_initialize
 * Description:
 *   Configure the IRQ subsystem
 ****************************************************************************/
void irq_initialize(void)
{
  /* Point all interrupt vectors to the unexpected interrupt */
  for (i = 0; i < NR_IRQS; i++)
    {
      g_irqvector[i].handler = irq_unexpected_isr;
    }
  up_irqinitialize();
}

Unexpected Interrupt Handler irq_unexpected_isr is called when an Interrupt is triggered and there's no Interrupt Handler attached to the Interrupt: sched/irq/irq_unexpectedisr.c

/****************************************************************************
 * Name: irq_unexpected_isr
 * Description:
 *   An interrupt has been received for an IRQ that was never registered
 *   with the system.
 ****************************************************************************/
int irq_unexpected_isr(int irq, FAR void *context, FAR void *arg)
{
  up_irq_save();
  _err("ERROR irq: %d\n", irq);
  PANIC();

To attach an Interrupt Handler, we set the Handler and the Argument in the Interrupt Vector Table: sched/irq/irq_attach.c

/****************************************************************************
 * Name: irq_attach
 * Description:
 *   Configure the IRQ subsystem so that IRQ number 'irq' is dispatched to
 *   'isr'
 ****************************************************************************/
int irq_attach(int irq, xcpt_t isr, FAR void *arg)
{
  ...
  /* Save the new ISR and its argument in the table. */
  g_irqvector[irq].handler = isr;
  g_irqvector[irq].arg     = arg;

When an Interrupt is triggered...

1. Arm CPU looks up the Arm64 Vector Table _vector_table: arch/arm64/src/common/arm64_vector_table.S

   /* Four types of exceptions:
   * - synchronous: aborts from MMU, SP/CP alignment checking, unallocated
   *   instructions, SVCs/SMCs/HVCs, ...)
   * - IRQ: group 1 (normal) interrupts
   * - FIQ: group 0 or secure interrupts
   * - SError: fatal system errors
   *
   * Four different contexts:
   * - from same exception level, when using the SP_EL0 stack pointer
   * - from same exception level, when using the SP_ELx stack pointer
   * - from lower exception level, when this is AArch64
   * - from lower exception level, when this is AArch32
   *
   * +------------------+------------------+-------------------------+
   * |     Address      |  Exception type  |       Description       |
   * +------------------+------------------+-------------------------+
   * | VBAR_ELn + 0x000 | Synchronous      | Current EL with SP0     |
   * |          + 0x080 | IRQ / vIRQ       |                         |
   * |          + 0x100 | FIQ / vFIQ       |                         |
   * |          + 0x180 | SError / vSError |                         |
   * +------------------+------------------+-------------------------+
   * |          + 0x200 | Synchronous      | Current EL with SPx     |
   * |          + 0x280 | IRQ / vIRQ       |                         |
   * |          + 0x300 | FIQ / vFIQ       |                         |
   * |          + 0x380 | SError / vSError |                         |
   * +------------------+------------------+-------------------------+
   * |          + 0x400 | Synchronous      | Lower EL using  AArch64 |
   * |          + 0x480 | IRQ / vIRQ       |                         |
   * |          + 0x500 | FIQ / vFIQ       |                         |
   * |          + 0x580 | SError / vSError |                         |
   * +------------------+------------------+-------------------------+
   * |          + 0x600 | Synchronous      | Lower EL using AArch64  |
   * |          + 0x680 | IRQ / vIRQ       |                         |
   * |          + 0x700 | FIQ / vFIQ       |                         |
   * |          + 0x780 | SError / vSError |                         |
   * +------------------+------------------+-------------------------+
   */
   GTEXT(_vector_table)
   SECTION_SUBSEC_FUNC(exc_vector_table,_vector_table_section,_vector_table)
       ...
       /* Current EL with SP0 / IRQ */
       .align 7
       arm64_enter_exception x0, x1
       b    arm64_irq_handler
       ...
       /* Current EL with SPx / IRQ */
       .align 7
       arm64_enter_exception x0, x1
       b    arm64_irq_handler
   

   (arm64_enter_exception is defined here)

2. Based on the Arm64 Vector Table _vector_table, Arm CPU jumps to arm64_irq_handler: arch/arm64/src/common/arm64_vectors.S

   /****************************************************************************
   * Name: arm64_irq_handler
   * Description:
   *   Interrupt exception handler
   ****************************************************************************/
   GTEXT(arm64_irq_handler)
   SECTION_FUNC(text, arm64_irq_handler)
       ...
       /* Call arm64_decodeirq() on the interrupt stack
       * with interrupts disabled
       */
       bl     arm64_decodeirq
   

3. arm64_irq_handler calls arm64_decodeirq to decode the Interrupt: arch/arm64/src/common/arm64_gicv3.c

   /***************************************************************************
   * Name: arm64_decodeirq
   * Description:
   *   This function is called from the IRQ vector handler in arm64_vectors.S.
   *   At this point, the interrupt has been taken and the registers have
   *   been saved on the stack.  This function simply needs to determine the
   *   the irq number of the interrupt and then to call arm_doirq to dispatch
   *   the interrupt.
   *  Input Parameters:
   *   regs - A pointer to the register save area on the stack.
   ***************************************************************************/
   // Decode IRQ for PinePhone, based on arm_decodeirq in arm_gicv2.c
   uint64_t * arm64_decodeirq(uint64_t * regs)
   {
     ...
     if (irq < NR_IRQS)
       {
         /* Dispatch the interrupt */
   
         regs = arm64_doirq(irq, regs);
   

4. arm64_decodeirq calls arm64_doirq to dispatch the Interrupt: arch/arm64/src/common/arm64_doirq.c

   /****************************************************************************
    * Name: arm64_doirq
   * Description:
   *   Receives the decoded GIC interrupt information and dispatches control
   *   to the attached interrupt handler.
   *
   ****************************************************************************/
   uint64_t *arm64_doirq(int irq, uint64_t * regs)
   {
     ...
     /* Deliver the IRQ */
     irq_dispatch(irq, regs);
   

5. irq_dispatch calls the Interrupt Handler fetched from the Interrupt Vector Table: sched/irq/irq_dispatch.c

   /****************************************************************************
    * Name: irq_dispatch
   * Description:
   *   This function must be called from the architecture-specific logic in
   *   order to dispatch an interrupt to the appropriate, registered handling
   *   logic.
   ****************************************************************************/
   void irq_dispatch(int irq, FAR void *context)
   {
     if ((unsigned)irq < NR_IRQS)
       {
         if (g_irqvector[ndx].handler)
           {
             vector = g_irqvector[ndx].handler;
             arg    = g_irqvector[ndx].arg;
           }
       }
     /* Then dispatch to the interrupt handler */
     CALL_VECTOR(ndx, vector, irq, context, arg);
   

How is the Arm64 Vector Table _vector_table configured in the Arm CPU?

The Arm64 Vector Table _vector_table is configured in the Arm CPU during EL1 Init by arm64_boot_el1_init: arch/arm64/src/common/arm64_boot.c

void arm64_boot_el1_init(void)
{
  /* Setup vector table */
  write_sysreg((uint64_t)_vector_table, vbar_el1);
  ARM64_ISB();

vbar_el1 refers to Vector Base Address Register EL1.

(See Arm Cortex-A53 Technical Reference Manual, page 4-121, "Vector Base Address Register, EL1")

(Arm64 Vector Table is also configured during EL3 Init by arm64_boot_el3_init)

EL1 Init arm64_boot_el1_init is called by our Startup Code: arch/arm64/src/common/arm64_head.S

    PRINT(switch_el1, "- Boot from EL1\r\n")

    /* EL1 init */
    bl    arm64_boot_el1_init

    /* set SP_ELx and Enable SError interrupts */
    msr   SPSel, #1
    msr   DAIFClr, #(DAIFCLR_ABT_BIT)
    isb

jump_to_c_entry:
    PRINT(jump_to_c_entry, "- Boot to C runtime for OS Initialize\r\n")
    ret x25

What are EL1 and EL3?

According to Arm Cortex-A53 Technical Reference Manual page 3-5 ("Exception Level")...

The ARMv8 exception model defines exception levels EL0-EL3, where:

• EL0 has the lowest software execution privilege, and execution at EL0 is called unprivileged execution.

• Increased exception levels, from 1 to 3, indicate increased software execution privilege.

• EL2 provides support for processor virtualization.

• EL3 provides support for a secure state, see Security state on page 3-6.

PinePhone only uses EL1 and EL2 (but not EL3)...

HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize

From this we see that NuttX runs mostly in EL1.

(EL1 is less privileged than EL2, which supports Processor Virtualization)

Dump Interrupt Vector Table

This is how we dump the Interrupt Vector Table to troubleshoot Interrupts...

Based on arch/arm64/src/common/arm64_arch_timer.c

#include "irq/irq.h" // For dumping Interrupt Vector Table

void up_timer_initialize(void)
{
  ...
  // Attach System Timer Interrupt Handler
  irq_attach(ARM_ARCH_TIMER_IRQ, arm64_arch_timer_compare_isr, 0);

  // Begin dumping Interrupt Vector Table
  sinfo("ARM_ARCH_TIMER_IRQ=%d\n", ARM_ARCH_TIMER_IRQ);
  sinfo("arm64_arch_timer_compare_isr=%p\n", arm64_arch_timer_compare_isr);
  sinfo("irq_unexpected_isr=%p\n", irq_unexpected_isr);
  for (int i = 0; i < NR_IRQS; i++)
    {
      sinfo("g_irqvector[%d].handler=%p\n", i, g_irqvector[i].handler);
    }
  // End dumping Interrupt Vector Table

This code runs at startup to attach the very first Interrupt Handler, for the System Timer Interrupt.

We see that the System Timer Interrupt Number (IRQ) is 27...

up_timer_initialize: ARM_ARCH_TIMER_IRQ=27
up_timer_initialize: arm64_arch_timer_compare_isr=0x4009ae18
up_timer_initialize: irq_unexpected_isr=0x400820e0

up_timer_initialize: g_irqvector[0].handler=0x400820e0
...
up_timer_initialize: g_irqvector[26].handler=0x400820e0
up_timer_initialize: g_irqvector[27].handler=0x4009ae18
up_timer_initialize: g_irqvector[28].handler=0x400820e0
...
up_timer_initialize: g_irqvector[219].handler=0x400820e0

All entries in the Interrupt Vector Table point to the Unexpected Interrupt Handler irq_unexpected_isr, except for g_irqvector[27] which points to the System Timer Interrupt Handler arm64_arch_timer_compare_isr.

Interrupt Debugging

Can we debug the Arm64 Interrupt Handler?

Yep we can write to the UART Port like this...

Based on arch/arm64/src/common/arm64_vectors.S

# PinePhone Allwinner A64 UART0 Base Address
#define UART1_BASE_ADDRESS 0x01C28000

# QEMU UART Base Address
# Previously: #define UART1_BASE_ADDRESS 0x9000000

/****************************************************************************
 * Name: arm64_irq_handler
 * Description:
 *   Interrupt exception handler
 ****************************************************************************/
GTEXT(arm64_irq_handler)
SECTION_FUNC(text, arm64_irq_handler)

    mov   x0, #84                 /* For Debug: 'T' */
    ldr   x1, =UART1_BASE_ADDRESS /* For Debug */
    strb  w0, [x1]                /* For Debug */

    /* switch to IRQ stack and save current sp on it. */
    ...

This will print "T" on the console whenever the Arm64 CPU triggers an Interrupt. (Assuming that the UART Buffer hasn't overflowed)

We can insert this debug code for every handler in arch/arm64/src/common/arm64_vectors.S...

• arm64_sync_exc: Handle synchronous exception

• arm64_irq_handler: Interrupt exception handler

• arm64_serror_handler: SError handler (Fatal System Errors)

• arm64_mode32_error: Mode32 Error

• arm64_irq_spurious: Spurious Interrupt

This is how we insert the debug code for every handler in arm64_vectors.S: https://gist.github.com/lupyuen/4bea83c61704080f1af18abfda63c77e

We can do the same for the Arm64 Vector Table: arch/arm64/src/common/arm64_vector_table.S

# PinePhone Allwinner A64 UART0 Base Address
#define UART1_BASE_ADDRESS 0x01C28000

# QEMU UART Base Address
# Previously: #define UART1_BASE_ADDRESS 0x9000000

/* Save Corruptible Registers and exception context
 * on the task stack
 * note: allocate stackframe with XCPTCONTEXT_GP_REGS
 *     which is ARM64_ESF_REGS + ARM64_CS_REGS
 *     but only save ARM64_ESF_REGS
 */
.macro arm64_enter_exception xreg0, xreg1
    sub    sp, sp, #8 * XCPTCONTEXT_GP_REGS

    stp    x0,  x1,  [sp, #8 * REG_X0]
    stp    x2,  x3,  [sp, #8 * REG_X2]
    ...
    stp    x28, x29, [sp, #8 * REG_X28]

    mov   x0, #88                 /* For Debug: 'X' */
    ldr   x1, =UART1_BASE_ADDRESS /* For Debug */
    strb  w0, [x1]                /* For Debug */

Memory Map

PinePhone depends on Arm's Memory Management Unit (MMU). We defined two MMU Memory Regions for PinePhone: RAM and Device I/O: arch/arm64/include/qemu/chip.h

// PinePhone Generic Interrupt Controller
// GIC_DIST:  0x01C80000 + 0x1000
// GIC_CPUIF: 0x01C80000 + 0x2000
#define CONFIG_GICD_BASE          0x01C81000  
#define CONFIG_GICR_BASE          0x01C82000  

// Previously:
// #define CONFIG_GICD_BASE          0x8000000
// #define CONFIG_GICR_BASE          0x80a0000

// PinePhone RAM: 0x4000 0000 to 0x4800 0000
#define CONFIG_RAMBANK1_ADDR      0x40000000
#define CONFIG_RAMBANK1_SIZE      MB(128)

// PinePhone Device I/O: 0x0 to 0x2000 0000
#define CONFIG_DEVICEIO_BASEADDR  0x00000000
#define CONFIG_DEVICEIO_SIZE      MB(512)

// Previously:
// #define CONFIG_DEVICEIO_BASEADDR  0x7000000
// #define CONFIG_DEVICEIO_SIZE      MB(512)

// PinePhone uboot load address (kernel_addr_r)
#define CONFIG_LOAD_BASE          0x40080000
// Previously: #define CONFIG_LOAD_BASE          0x40280000

We also changed CONFIG_LOAD_BASE for PinePhone's Kernel Start Address (kernel_addr_r).

How are the MMU Memory Regions used?

NuttX initialises the Arm MMU with the MMU Memory Regions at startup: arch/arm64/src/qemu/qemu_boot.c

static const struct arm_mmu_region mmu_regions[] =
{
  MMU_REGION_FLAT_ENTRY("DEVICE_REGION",
                        CONFIG_DEVICEIO_BASEADDR, MB(512),
                        MT_DEVICE_NGNRNE | MT_RW | MT_SECURE),

  MMU_REGION_FLAT_ENTRY("DRAM0_S0",
                        CONFIG_RAMBANK1_ADDR, MB(512),
                        MT_NORMAL | MT_RW | MT_SECURE),
};

const struct arm_mmu_config mmu_config =
{
  .num_regions = ARRAY_SIZE(mmu_regions),
  .mmu_regions = mmu_regions,
};

The Arm MMU Initialisation is done by arm64_mmu_init, defined in arch/arm64/src/common/arm64_mmu.c

We'll talk more about the Arm MMU in the next section...

Boot Sequence

This section describes the Boot Sequence for NuttX on PinePhone.

The Startup Code (in Arm64 Assembly) inits the Arm64 System Registers, UART Port and jumps to arm64_boot_secondary_c_routine (in C): arch/arm64/src/common/arm64_head.S

    ldr    x25, =arm64_boot_secondary_c_routine
    ...
jump_to_c_entry:
    PRINT(jump_to_c_entry, "- Boot to C runtime for OS Initialize\r\n")
    ret x25

arm64_boot_primary_c_routine inits the BSS, calls arm64_chip_boot to init the Arm64 CPU, and nx_start to start the NuttX processes: arch/arm64/src/common/arm64_boot.c

void arm64_boot_primary_c_routine(void)
{
  boot_early_memset(_START_BSS, 0, _END_BSS - _START_BSS);
  arm64_chip_boot();
  nx_start();
}

arm64_chip_boot calls arm64_mmu_init to enable the Arm Memory Management Unit, and qemu_board_initialize to init the Board Drivers: arch/arm64/src/qemu/qemu_boot.c

void arm64_chip_boot(void)
{
  /* MAP IO and DRAM, enable MMU. */

  arm64_mmu_init(true);

#ifdef CONFIG_SMP
  arm64_psci_init("smc");

#endif

  /* Perform board-specific device initialization. This would include
   * configuration of board specific resources such as GPIOs, LEDs, etc.
   */

  qemu_board_initialize();

#ifdef USE_EARLYSERIALINIT
  /* Perform early serial initialization if we are going to use the serial
   * driver.
   */

  qemu_earlyserialinit();
#endif
}

arm64_mmu_init is defined in arch/arm64/src/common/arm64_mmu.c

The next section talks about debugging the Boot Sequence...

Boot Debugging

How can we debug NuttX while it boots?

We may call up_putc to print characters to the Serial Console and troubleshoot the Boot Sequence: arch/arm64/src/common/arm64_boot.c

void arm64_boot_primary_c_routine(void)
{
  int up_putc(int ch);  // For debugging
  up_putc('0');  // For debugging
  boot_early_memset(_START_BSS, 0, _END_BSS - _START_BSS);
  up_putc('1');  // For debugging
  arm64_chip_boot();
  up_putc('2');  // For debugging
  nx_start();
}

This prints "012" to the Serial Console as NuttX boots.

up_putc calls up_lowputc to print directly to the UART Port by writing to the UART Register. So it's safe to be called as NuttX boots.

GIC Register Dump

Below is the dump of PinePhone's registers for Arm Generic Interrupt Controller version 2...

HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x400c4000, heap_size=0x7f3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: GIC Version is 2
Earm_gic_dump: GIC: Entry arm_gic0_initialize NLINES=224
arm_gic_dump_cpu:   CPU Interface Registers:
arm_gic_dump_cpu:        ICR: 00000060    PMR: 000000f0    BPR: 00000003    IAR: 000003ff
arm_gic_dump_cpu:        RPR: 000000ff   HPIR: 000003ff   ABPR: 00000000
arm_gic_dump_cpu:       AIAR: 00000000  AHPIR: 00000000    IDR: 0202143b
arm_gic_dump_cpu:       APR1: 00000000   APR2: 00000000   APR3: 00000000   APR4: 00000000
arm_gic_dump_cpu:     NSAPR1: 00000000 NSAPR2: 00000000 NSAPR3: 00000000 NSAPR4: 00000000
arm_gic_dump_distributor:   Distributor Registers:
arm_gic_dump_distributor:        DCR: 00000000   ICTR: 0000fc66   IIDR: 0200143b
arm_gic_dump32:        ISR[01c81080]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        ISER/ICER[01c81100]
arm_gic_dumpregs:          0000ffff 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        ISPR/ICPR[01c81200]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        SAR/CAR[01c81300]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump4:        IPR[01c81400]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump4:        IPTR[01c81800]
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          00000000 00000000 01010100 01010101
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump16:        ICFR[01c81c00]
arm_gic_dumpregs:          aaaaaaaa 55540000 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 00000000 00000000
arm_gic_dump32:        PPSIR/SPISR[01c81d00]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        NSACR[01c81e00]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump8:        SCPR/SSPR[01c81f10]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump_distributor:        PIDR[01c81fd0]:
arm_gic_dump_distributor:          00000004 00000000 00000000 00000000
arm_gic_dump_distributor:          00000090 000000b4 0000002b
arm_gic_dump_distributor:        CIDR[01c81ff0]:
arm_gic_dump_distributor:          0000000d 000000f0 00000005 000000b1
arm_gic_dump: GIC: Exit arm_gic0_initialize NLINES=224
arm_gic_dump_cpu:   CPU Interface Registers:
arm_gic_dump_cpu:        ICR: 00000060    PMR: 000000f0    BPR: 00000003    IAR: 000003ff
arm_gic_dump_cpu:        RPR: 000000ff   HPIR: 000003ff   ABPR: 00000000
arm_gic_dump_cpu:       AIAR: 00000000  AHPIR: 00000000    IDR: 0202143b
arm_gic_dump_cpu:       APR1: 00000000   APR2: 00000000   APR3: 00000000   APR4: 00000000
arm_gic_dump_cpu:     NSAPR1: 00000000 NSAPR2: 00000000 NSAPR3: 00000000 NSAPR4: 00000000
arm_gic_dump_distributor:   Distributor Registers:
arm_gic_dump_distributor:        DCR: 00000000   ICTR: 0000fc66   IIDR: 0200143b
arm_gic_dump32:        ISR[01c81080]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        ISER/ICER[01c81100]
arm_gic_dumpregs:          0000ffff 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        ISPR/ICPR[01c81200]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        SAR/CAR[01c81300]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump4:        IPR[01c81400]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dump4:        IPTR[01c81800]
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          00000000 00000000 01010100 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dump16:        ICFR[01c81c00]
arm_gic_dumpregs:          aaaaaaaa 55540000 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 00000000 00000000
arm_gic_dump32:        PPSIR/SPISR[01c81d00]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        NSACR[01c81e00]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump8:        SCPR/SSPR[01c81f10]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump_distributor:        PIDR[01c81fd0]:
arm_gic_dump_distributor:          00000004 00000000 00000000 00000000
arm_gic_dump_distributor:          00000090 000000b4 0000002b
arm_gic_dump_distributor:        CIDR[01c81ff0]:
arm_gic_dump_distributor:          0000000d 000000f0 00000005 000000b1
FGarm_gic_dump: GIC: Entry arm_gic_initialize NLINES=224
arm_gic_dump_cpu:   CPU Interface Registers:
arm_gic_dump_cpu:        ICR: 00000060    PMR: 000000f0    BPR: 00000003    IAR: 000003ff
arm_gic_dump_cpu:        RPR: 000000ff   HPIR: 000003ff   ABPR: 00000000
arm_gic_dump_cpu:       AIAR: 00000000  AHPIR: 00000000    IDR: 0202143b
arm_gic_dump_cpu:       APR1: 00000000   APR2: 00000000   APR3: 00000000   APR4: 00000000
arm_gic_dump_cpu:     NSAPR1: 00000000 NSAPR2: 00000000 NSAPR3: 00000000 NSAPR4: 00000000
arm_gic_dump_distributor:   Distributor Registers:
arm_gic_dump_distributor:        DCR: 00000000   ICTR: 0000fc66   IIDR: 0200143b
arm_gic_dump32:        ISR[01c81080]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        ISER/ICER[01c81100]
arm_gic_dumpregs:          0000ffff 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        ISPR/ICPR[01c81200]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        SAR/CAR[01c81300]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump4:        IPR[01c81400]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dump4:        IPTR[01c81800]
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          00000000 00000000 01010100 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dump16:        ICFR[01c81c00]
arm_gic_dumpregs:          aaaaaaaa 55540000 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 00000000 00000000
arm_gic_dump32:        PPSIR/SPISR[01c81d00]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        NSACR[01c81e00]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump8:        SCPR/SSPR[01c81f10]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump_distributor:        PIDR[01c81fd0]:
arm_gic_dump_distributor:          00000004 00000000 00000000 00000000
arm_gic_dump_distributor:          00000090 000000b4 0000002b
arm_gic_dump_distributor:        CIDR[01c81ff0]:
arm_gic_dump_distributor:          0000000d 000000f0 00000005 000000b1
arm_gic_dump: GIC: Exit arm_gic_initialize NLINES=224
arm_gic_dump_cpu:   CPU Interface Registers:
arm_gic_dump_cpu:        ICR: 00000061    PMR: 000000f0    BPR: 00000007    IAR: 000003ff
arm_gic_dump_cpu:        RPR: 000000ff   HPIR: 000003ff   ABPR: 00000000
arm_gic_dump_cpu:       AIAR: 00000000  AHPIR: 00000000    IDR: 0202143b
arm_gic_dump_cpu:       APR1: 00000000   APR2: 00000000   APR3: 00000000   APR4: 00000000
arm_gic_dump_cpu:     NSAPR1: 00000000 NSAPR2: 00000000 NSAPR3: 00000000 NSAPR4: 00000000
arm_gic_dump_distributor:   Distributor Registers:
arm_gic_dump_distributor:        DCR: 00000001   ICTR: 0000fc66   IIDR: 0200143b
arm_gic_dump32:        ISR[01c81080]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        ISER/ICER[01c81100]
arm_gic_dumpregs:          0000ffff 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        ISPR/ICPR[01c81200]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        SAR/CAR[01c81300]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump4:        IPR[01c81400]
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          00000000 00000000 80000000 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dumpregs:          80808080 80808080 80808080 80808080
arm_gic_dump4:        IPTR[01c81800]
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          00000000 00000000 01010100 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dumpregs:          01010101 01010101 01010101 01010101
arm_gic_dump16:        ICFR[01c81c00]
arm_gic_dumpregs:          aaaaaaaa 55540000 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 55555555 55555555
arm_gic_dumpregs:          55555555 55555555 00000000 00000000
arm_gic_dump32:        PPSIR/SPISR[01c81d00]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump32:        NSACR[01c81e00]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump8:        SCPR/SSPR[01c81f10]
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dumpregs:          00000000 00000000 00000000 00000000
arm_gic_dump_distributor:        PIDR[01c81fd0]:
arm_gic_dump_distributor:          00000004 00000000 00000000 00000000
arm_gic_dump_distributor:          00000090 000000b4 0000002b
arm_gic_dump_distributor:        CIDR[01c81ff0]:
arm_gic_dump_distributor:          0000000d 000000f0 00000005 000000b1
Hup_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
AMarm_gic_dump: GIC: Exit up_prioritize_irq IRQ=27
arm_gic_dump_cpu:   CPU Interface Registers:
arm_gic_dump_cpu:        ICR: 00000061    PMR: 000000f0    BPR: 00000007    IAR: 000003ff
arm_gic_dump_cpu:        RPR: 000000ff   HPIR: 000003ff   ABPR: 00000000
arm_gic_dump_cpu:       AIAR: 00000000  AHPIR: 00000000    IDR: 0202143b
arm_gic_dump_cpu:       APR1: 00000000   APR2: 00000000   APR3: 00000000   APR4: 00000000
arm_gic_dump_cpu:     NSAPR1: 00000000 NSAPR2: 00000000 NSAPR3: 00000000 NSAPR4: 00000000
arm_gic_dump_distributor:   Distributor Registers:
arm_gic_dump_distributor:        DCR: 00000001   ICTR: 0000fc66   IIDR: 0200143b
arm_gic_dump_distributor:        ISR: 00000000   ISER: 0000ffff   ISPR: 00000000    SAR: 00000000
arm_gic_dump_distributor:        IPR: a0000000   IPTR: 01010100   ICFR: 55540000  SPISR: 00000000
arm_gic_dump_distributor:      NSACR: 00000000   SCPR: 00000000
arm_gic_dump_distributor:        PIDR[01c81fd0]:
arm_gic_dump_distributor:          00000004 00000000 00000000 00000000
arm_gic_dump_distributor:          00000090 000000b4 0000002b
arm_gic_dump_distributor:        CIDR[01c81ff0]:
arm_gic_dump_distributor:          0000000d 000000f0 00000005 000000b1
NOPBIarm_gic_du

TODO

TODO: sinfo (syslog) works, but printf (puts) doesn't!

https://github.com/lupyuen/incubator-nuttx-apps/blob/pinephone/system/nsh/nsh_main.c#L88-L102

/****************************************************************************
 * Name: nsh_main
 *
 * Description:
 *   This is the main logic for the case of the NSH task.  It will perform
 *   one-time NSH initialization and start an interactive session on the
 *   current console device.
 *
 ****************************************************************************/

int main(int argc, FAR char *argv[])
{
  sinfo("****main\n");////
  printf("****main2\n");////
  sinfo("****main3\n");////

PinePhone:

HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
L: qemu_pl011_setup
nx_start: Entry
up_allocate_heap: heap_start=0x0x400c4000, heap_size=0x7f3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
up_timer_initialize: _vector_table=0x400a7000
up_timer_initialize: Before writing: vbar_el1=0x40227000
up_timer_initialize: After writing: vbar_el1=0x400a7000
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
K: qemu_pl011_attach
G: qemu_pl011_rxint
work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x400a7000 _einit: 0x400a7000 _stext: 0x40080000 _etext: 0x400a8000
nsh_main: ****main
puts: A
HH: qemu_pl011_txint
puts: B
nsh_main: ****main3
HHHHHHHHHHHH: qemu_pl011_txint
GG: qemu_pl011_rxint
nx_start: CPU0: Beginning Idle Loop

QEMU:

- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x402c4000, heap_size=0x7d3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x8000000
arm64_gic_initialize: CONFIG_GICR_BASE=0x8010000
arm64_gic_initialize: GIC Version is 2
EFGHup_timer_initialize: up_timer_initialize: cp15 timer(s) running at 62.50MHz, cycle 62500
up_timer_initialize: ARM_ARCH_TIMER_IRQ=27
up_timer_initialize: arm64_arch_timer_compare_isr=0x4029b2ac
up_timer_initialize: irq_unexpected_isr=0x402823ec
up_timer_initialize: g_irqvector[0].handler=0x402823ec
up_timer_initialize: g_irqvector[1].handler=0x402823ec
up_timer_initialize: g_irqvector[2].handler=0x402823ec
up_timer_initialize: g_irqvector[3].handler=0x402823ec
up_timer_initialize: g_irqvector[4].handler=0x402823ec
up_timer_initialize: g_irqvector[5].handler=0x402823ec
up_timer_initialize: g_irqvector[6].handler=0x402823ec
up_timer_initialize: g_irqvector[7].handler=0x402823ec
up_timer_initialize: g_irqvector[8].handler=0x402823ec
up_timer_initialize: g_irqvector[9].handler=0x402823ec
up_timer_initialize: g_irqvector[10].handler=0x402823ec
up_timer_initialize: g_irqvector[11].handler=0x402823ec
up_timer_initialize: g_irqvector[12].handler=0x402823ec
up_timer_initialize: g_irqvector[13].handler=0x402823ec
up_timer_initialize: g_irqvector[14].handler=0x402823ec
up_timer_initialize: g_irqvector[15].handler=0x402823ec
up_timer_initialize: g_irqvector[16].handler=0x402823ec
up_timer_initialize: g_irqvector[17].handler=0x402823ec
up_timer_initialize: g_irqvector[18].handler=0x402823ec
up_timer_initialize: g_irqvector[19].handler=0x402823ec
up_timer_initialize: g_irqvector[20].handler=0x402823ec
up_timer_initialize: g_irqvector[21].handler=0x402823ec
up_timer_initialize: g_irqvector[22].handler=0x402823ec
up_timer_initialize: g_irqvector[23].handler=0x402823ec
up_timer_initialize: g_irqvector[24].handler=0x402823ec
up_timer_initialize: g_irqvector[25].handler=0x402823ec
up_timer_initialize: g_irqvector[26].handler=0x402823ec
up_timer_initialize: g_irqvector[27].handler=0x4029b2ac
up_timer_initialize: g_irqvector[28].handler=0x402823ec
up_timer_initialize: g_irqvector[29].handler=0x402823ec
up_timer_initialize: g_irqvector[30].handler=0x402823ec
up_timer_initialize: g_irqvector[31].handler=0x402823ec
up_timer_initialize: g_irqvector[32].handler=0x402823ec
up_timer_initialize: g_irqvector[33].handler=0x402823ec
up_timer_initialize: g_irqvector[34].handler=0x402823ec
up_timer_initialize: g_irqvector[35].handler=0x402823ec
up_timer_initialize: g_irqvector[36].handler=0x402823ec
up_timer_initialize: g_irqvector[37].handler=0x402823ec
up_timer_initialize: g_irqvector[38].handler=0x402823ec
up_timer_initialize: g_irqvector[39].handler=0x402823ec
up_timer_initialize: g_irqvector[40].handler=0x402823ec
up_timer_initialize: g_irqvector[41].handler=0x402823ec
up_timer_initialize: g_irqvector[42].handler=0x402823ec
up_timer_initialize: g_irqvector[43].handler=0x402823ec
up_timer_initialize: g_irqvector[44].handler=0x402823ec
up_timer_initialize: g_irqvector[45].handler=0x402823ec
up_timer_initialize: g_irqvector[46].handler=0x402823ec
up_timer_initialize: g_irqvector[47].handler=0x402823ec
up_timer_initialize: g_irqvector[48].handler=0x402823ec
up_timer_initialize: g_irqvector[49].handler=0x402823ec
up_timer_initialize: g_irqvector[50].handler=0x402823ec
up_timer_initialize: g_irqvector[51].handler=0x402823ec
up_timer_initialize: g_irqvector[52].handler=0x402823ec
up_timer_initialize: g_irqvector[53].handler=0x402823ec
up_timer_initialize: g_irqvector[54].handler=0x402823ec
up_timer_initialize: g_irqvector[55].handler=0x402823ec
up_timer_initialize: g_irqvector[56].handler=0x402823ec
up_timer_initialize: g_irqvector[57].handler=0x402823ec
up_timer_initialize: g_irqvector[58].handler=0x402823ec
up_timer_initialize: g_irqvector[59].handler=0x402823ec
up_timer_initialize: g_irqvector[60].handler=0x402823ec
up_timer_initialize: g_irqvector[61].handler=0x402823ec
up_timer_initialize: g_irqvector[62].handler=0x402823ec
up_timer_initialize: g_irqvector[63].handler=0x402823ec
up_timer_initialize: g_irqvector[64].handler=0x402823ec
up_timer_initialize: g_irqvector[65].handler=0x402823ec
up_timer_initialize: g_irqvector[66].handler=0x402823ec
up_timer_initialize: g_irqvector[67].handler=0x402823ec
up_timer_initialize: g_irqvector[68].handler=0x402823ec
up_timer_initialize: g_irqvector[69].handler=0x402823ec
up_timer_initialize: g_irqvector[70].handler=0x402823ec
up_timer_initialize: g_irqvector[71].handler=0x402823ec
up_timer_initialize: g_irqvector[72].handler=0x402823ec
up_timer_initialize: g_irqvector[73].handler=0x402823ec
up_timer_initialize: g_irqvector[74].handler=0x402823ec
up_timer_initialize: g_irqvector[75].handler=0x402823ec
up_timer_initialize: g_irqvector[76].handler=0x402823ec
up_timer_initialize: g_irqvector[77].handler=0x402823ec
up_timer_initialize: g_irqvector[78].handler=0x402823ec
up_timer_initialize: g_irqvector[79].handler=0x402823ec
up_timer_initialize: g_irqvector[80].handler=0x402823ec
up_timer_initialize: g_irqvector[81].handler=0x402823ec
up_timer_initialize: g_irqvector[82].handler=0x402823ec
up_timer_initialize: g_irqvector[83].handler=0x402823ec
up_timer_initialize: g_irqvector[84].handler=0x402823ec
up_timer_initialize: g_irqvector[85].handler=0x402823ec
up_timer_initialize: g_irqvector[86].handler=0x402823ec
up_timer_initialize: g_irqvector[87].handler=0x402823ec
up_timer_initialize: g_irqvector[88].handler=0x402823ec
up_timer_initialize: g_irqvector[89].handler=0x402823ec
up_timer_initialize: g_irqvector[90].handler=0x402823ec
up_timer_initialize: g_irqvector[91].handler=0x402823ec
up_timer_initialize: g_irqvector[92].handler=0x402823ec
up_timer_initialize: g_irqvector[93].handler=0x402823ec
up_timer_initialize: g_irqvector[94].handler=0x402823ec
up_timer_initialize: g_irqvector[95].handler=0x402823ec
up_timer_initialize: g_irqvector[96].handler=0x402823ec
up_timer_initialize: g_irqvector[97].handler=0x402823ec
up_timer_initialize: g_irqvector[98].handler=0x402823ec
up_timer_initialize: g_irqvector[99].handler=0x402823ec
up_timer_initialize: g_irqvector[100].handler=0x402823ec
up_timer_initialize: g_irqvector[101].handler=0x402823ec
up_timer_initialize: g_irqvector[102].handler=0x402823ec
up_timer_initialize: g_irqvector[103].handler=0x402823ec
up_timer_initialize: g_irqvector[104].handler=0x402823ec
up_timer_initialize: g_irqvector[105].handler=0x402823ec
up_timer_initialize: g_irqvector[106].handler=0x402823ec
up_timer_initialize: g_irqvector[107].handler=0x402823ec
up_timer_initialize: g_irqvector[108].handler=0x402823ec
up_timer_initialize: g_irqvector[109].handler=0x402823ec
up_timer_initialize: g_irqvector[110].handler=0x402823ec
up_timer_initialize: g_irqvector[111].handler=0x402823ec
up_timer_initialize: g_irqvector[112].handler=0x402823ec
up_timer_initialize: g_irqvector[113].handler=0x402823ec
up_timer_initialize: g_irqvector[114].handler=0x402823ec
up_timer_initialize: g_irqvector[115].handler=0x402823ec
up_timer_initialize: g_irqvector[116].handler=0x402823ec
up_timer_initialize: g_irqvector[117].handler=0x402823ec
up_timer_initialize: g_irqvector[118].handler=0x402823ec
up_timer_initialize: g_irqvector[119].handler=0x402823ec
up_timer_initialize: g_irqvector[120].handler=0x402823ec
up_timer_initialize: g_irqvector[121].handler=0x402823ec
up_timer_initialize: g_irqvector[122].handler=0x402823ec
up_timer_initialize: g_irqvector[123].handler=0x402823ec
up_timer_initialize: g_irqvector[124].handler=0x402823ec
up_timer_initialize: g_irqvector[125].handler=0x402823ec
up_timer_initialize: g_irqvector[126].handler=0x402823ec
up_timer_initialize: g_irqvector[127].handler=0x402823ec
up_timer_initialize: g_irqvector[128].handler=0x402823ec
up_timer_initialize: g_irqvector[129].handler=0x402823ec
up_timer_initialize: g_irqvector[130].handler=0x402823ec
up_timer_initialize: g_irqvector[131].handler=0x402823ec
up_timer_initialize: g_irqvector[132].handler=0x402823ec
up_timer_initialize: g_irqvector[133].handler=0x402823ec
up_timer_initialize: g_irqvector[134].handler=0x402823ec
up_timer_initialize: g_irqvector[135].handler=0x402823ec
up_timer_initialize: g_irqvector[136].handler=0x402823ec
up_timer_initialize: g_irqvector[137].handler=0x402823ec
up_timer_initialize: g_irqvector[138].handler=0x402823ec
up_timer_initialize: g_irqvector[139].handler=0x402823ec
up_timer_initialize: g_irqvector[140].handler=0x402823ec
up_timer_initialize: g_irqvector[141].handler=0x402823ec
up_timer_initialize: g_irqvector[142].handler=0x402823ec
up_timer_initialize: g_irqvector[143].handler=0x402823ec
up_timer_initialize: g_irqvector[144].handler=0x402823ec
up_timer_initialize: g_irqvector[145].handler=0x402823ec
up_timer_initialize: g_irqvector[146].handler=0x402823ec
up_timer_initialize: g_irqvector[147].handler=0x402823ec
up_timer_initialize: g_irqvector[148].handler=0x402823ec
up_timer_initialize: g_irqvector[149].handler=0x402823ec
up_timer_initialize: g_irqvector[150].handler=0x402823ec
up_timer_initialize: g_irqvector[151].handler=0x402823ec
up_timer_initialize: g_irqvector[152].handler=0x402823ec
up_timer_initialize: g_irqvector[153].handler=0x402823ec
up_timer_initialize: g_irqvector[154].handler=0x402823ec
up_timer_initialize: g_irqvector[155].handler=0x402823ec
up_timer_initialize: g_irqvector[156].handler=0x402823ec
up_timer_initialize: g_irqvector[157].handler=0x402823ec
up_timer_initialize: g_irqvector[158].handler=0x402823ec
up_timer_initialize: g_irqvector[159].handler=0x402823ec
up_timer_initialize: g_irqvector[160].handler=0x402823ec
up_timer_initialize: g_irqvector[161].handler=0x402823ec
up_timer_initialize: g_irqvector[162].handler=0x402823ec
up_timer_initialize: g_irqvector[163].handler=0x402823ec
up_timer_initialize: g_irqvector[164].handler=0x402823ec
up_timer_initialize: g_irqvector[165].handler=0x402823ec
up_timer_initialize: g_irqvector[166].handler=0x402823ec
up_timer_initialize: g_irqvector[167].handler=0x402823ec
up_timer_initialize: g_irqvector[168].handler=0x402823ec
up_timer_initialize: g_irqvector[169].handler=0x402823ec
up_timer_initialize: g_irqvector[170].handler=0x402823ec
up_timer_initialize: g_irqvector[171].handler=0x402823ec
up_timer_initialize: g_irqvector[172].handler=0x402823ec
up_timer_initialize: g_irqvector[173].handler=0x402823ec
up_timer_initialize: g_irqvector[174].handler=0x402823ec
up_timer_initialize: g_irqvector[175].handler=0x402823ec
up_timer_initialize: g_irqvector[176].handler=0x402823ec
up_timer_initialize: g_irqvector[177].handler=0x402823ec
up_timer_initialize: g_irqvector[178].handler=0x402823ec
up_timer_initialize: g_irqvector[179].handler=0x402823ec
up_timer_initialize: g_irqvector[180].handler=0x402823ec
up_timer_initialize: g_irqvector[181].handler=0x402823ec
up_timer_initialize: g_irqvector[182].handler=0x402823ec
up_timer_initialize: g_irqvector[183].handler=0x402823ec
up_timer_initialize: g_irqvector[184].handler=0x402823ec
up_timer_initialize: g_irqvector[185].handler=0x402823ec
up_timer_initialize: g_irqvector[186].handler=0x402823ec
up_timer_initialize: g_irqvector[187].handler=0x402823ec
up_timer_initialize: g_irqvector[188].handler=0x402823ec
up_timer_initialize: g_irqvector[189].handler=0x402823ec
up_timer_initialize: g_irqvector[190].handler=0x402823ec
up_timer_initialize: g_irqvector[191].handler=0x402823ec
up_timer_initialize: g_irqvector[192].handler=0x402823ec
up_timer_initialize: g_irqvector[193].handler=0x402823ec
up_timer_initialize: g_irqvector[194].handler=0x402823ec
up_timer_initialize: g_irqvector[195].handler=0x402823ec
up_timer_initialize: g_irqvector[196].handler=0x402823ec
up_timer_initialize: g_irqvector[197].handler=0x402823ec
up_timer_initialize: g_irqvector[198].handler=0x402823ec
up_timer_initialize: g_irqvector[199].handler=0x402823ec
up_timer_initialize: g_irqvector[200].handler=0x402823ec
up_timer_initialize: g_irqvector[201].handler=0x402823ec
up_timer_initialize: g_irqvector[202].handler=0x402823ec
up_timer_initialize: g_irqvector[203].handler=0x402823ec
up_timer_initialize: g_irqvector[204].handler=0x402823ec
up_timer_initialize: g_irqvector[205].handler=0x402823ec
up_timer_initialize: g_irqvector[206].handler=0x402823ec
up_timer_initialize: g_irqvector[207].handler=0x402823ec
up_timer_initialize: g_irqvector[208].handler=0x402823ec
up_timer_initialize: g_irqvector[209].handler=0x402823ec
up_timer_initialize: g_irqvector[210].handler=0x402823ec
up_timer_initialize: g_irqvector[211].handler=0x402823ec
up_timer_initialize: g_irqvector[212].handler=0x402823ec
up_timer_initialize: g_irqvector[213].handler=0x402823ec
up_timer_initialize: g_irqvector[214].handler=0x402823ec
up_timer_initialize: g_irqvector[215].handler=0x402823ec
up_timer_initialize: g_irqvector[216].handler=0x402823ec
up_timer_initialize: g_irqvector[217].handler=0x402823ec
up_timer_initialize: g_irqvector[218].handler=0x402823ec
up_timer_initialize: g_irqvector[219].handler=0x402823ec
AKLMNOPBIJQRQRuart_register: Registering /dev/console
QRuart_register: Registering /dev/ttyS0
QRAKLMNOPBIJQRQRQRwork_start_highpri: Starting high-priority kernel worker thread(s)
QRQRQRnx_start_application: StartinQRg init thread
QRQRQRlib_cxx_initialize: _sinit: 0x402a7000 _einit: 0x402a700QR0 _stext: 0x40280000 _etext: 0x402a8000
QRQRQRQRQRQRQRQnsh: sysinit: fopen failed: 2
QRQRQRQRQRQRQRQRQRQRQRQRQnsh: mkfatfs: command not found
QRQRQRQRQ
QNuttShell (NSH) NuttX-10.3.0-RC2
Qnsh> QQRnx_start: CPU0: Beginning Idle Loop
QRQRQRQRRRRRRR

TODO: Boot Files for Manjaro Phosh on PinePhone:

[manjaro@manjaro-arm ~]$ ls -l /boot
total 38568
-rw-r--r-- 1 root root     1476 Jun 22 08:36 boot.scr
-rw-r--r-- 1 root root     1404 Apr  6 11:51 boot.txt
drwxr-xr-x 3 root root     4096 Oct 16  2021 dtbs
-rw-r--r-- 1 root root 20160520 Jul  3 14:56 Image
-rw-r--r-- 1 root root  8359044 Jul  3 14:56 Image.gz
-rw-r--r-- 1 root root  7327835 Jul 24 14:33 initramfs-linux.img
-rw-r--r-- 1 root root   722223 Apr  6 11:51 u-boot-sunxi-with-spl-pinephone-492.bin
-rw-r--r-- 1 root root   722223 Apr  6 11:51 u-boot-sunxi-with-spl-pinephone-528.bin
-rw-r--r-- 1 root root   722223 Apr  6 11:51 u-boot-sunxi-with-spl-pinephone-552.bin
-rw-r--r-- 1 root root   722223 Apr  6 11:51 u-boot-sunxi-with-spl-pinephone-592.bin
-rw-r--r-- 1 root root   722223 Apr  6 11:51 u-boot-sunxi-with-spl-pinephone-624.bin

[manjaro@manjaro-arm ~]$ ls -l /boot/dtbs
total 8
drwxr-xr-x 2 root root 8192 Jul 24 14:30 allwinner

[manjaro@manjaro-arm ~]$ ls -l /boot/dtbs/allwinner
total 1504
-rw-r--r-- 1 root root 13440 Jul  3 14:56 sun50i-a100-allwinner-perf1.dtb
-rw-r--r-- 1 root root 41295 Jul  3 14:56 sun50i-a64-amarula-relic.dtb
-rw-r--r-- 1 root root 41648 Jul  3 14:56 sun50i-a64-bananapi-m64.dtb
-rw-r--r-- 1 root root 40512 Jul  3 14:56 sun50i-a64-nanopi-a64.dtb
-rw-r--r-- 1 root root 39951 Jul  3 14:56 sun50i-a64-oceanic-5205-5inmfd.dtb
-rw-r--r-- 1 root root 41268 Jul  3 14:56 sun50i-a64-olinuxino.dtb
-rw-r--r-- 1 root root 41397 Jul  3 14:56 sun50i-a64-olinuxino-emmc.dtb
-rw-r--r-- 1 root root 42295 Jul  3 14:56 sun50i-a64-orangepi-win.dtb
-rw-r--r-- 1 root root 40316 Jul  3 14:56 sun50i-a64-pine64.dtb
-rw-r--r-- 1 root root 40948 Jul  3 14:56 sun50i-a64-pine64-lts.dtb
-rw-r--r-- 1 root root 40438 Jul  3 14:56 sun50i-a64-pine64-plus.dtb
-rw-r--r-- 1 root root 42979 Jul  3 14:56 sun50i-a64-pinebook.dtb
-rw-r--r-- 1 root root 53726 Jul  3 14:56 sun50i-a64-pinephone-1.0.dtb
-rw-r--r-- 1 root root 53753 Jul  3 14:56 sun50i-a64-pinephone-1.1.dtb
-rw-r--r-- 1 root root 53718 Jul  3 14:56 sun50i-a64-pinephone-1.2.dtb
-rw-r--r-- 1 root root 44110 Jul  3 14:56 sun50i-a64-pinetab.dtb
-rw-r--r-- 1 root root 44150 Jul  3 14:56 sun50i-a64-pinetab-early-adopter.dtb
-rw-r--r-- 1 root root 40816 Jul  3 14:56 sun50i-a64-sopine-baseboard.dtb
-rw-r--r-- 1 root root 42234 Jul  3 14:56 sun50i-a64-teres-i.dtb
-rw-r--r-- 1 root root 31407 Jul  3 14:56 sun50i-h5-bananapi-m2-plus.dtb
-rw-r--r-- 1 root root 32846 Jul  3 14:56 sun50i-h5-bananapi-m2-plus-v1.2.dtb
-rw-r--r-- 1 root root 31056 Jul  3 14:56 sun50i-h5-emlid-neutis-n5-devboard.dtb
-rw-r--r-- 1 root root 31277 Jul  3 14:56 sun50i-h5-libretech-all-h3-cc.dtb
-rw-r--r-- 1 root root 29939 Jul  3 14:56 sun50i-h5-libretech-all-h3-it.dtb
-rw-r--r-- 1 root root 31872 Jul  3 14:56 sun50i-h5-libretech-all-h5-cc.dtb
-rw-r--r-- 1 root root 29013 Jul  3 14:56 sun50i-h5-nanopi-neo2.dtb
-rw-r--r-- 1 root root 29704 Jul  3 14:56 sun50i-h5-nanopi-neo-plus2.dtb
-rw-r--r-- 1 root root 31401 Jul  3 14:56 sun50i-h5-nanopi-r1s-h5.dtb
-rw-r--r-- 1 root root 31082 Jul  3 14:56 sun50i-h5-orangepi-pc2.dtb
-rw-r--r-- 1 root root 29806 Jul  3 14:56 sun50i-h5-orangepi-prime.dtb
-rw-r--r-- 1 root root 29044 Jul  3 14:56 sun50i-h5-orangepi-zero-plus2.dtb
-rw-r--r-- 1 root root 29131 Jul  3 14:56 sun50i-h5-orangepi-zero-plus.dtb
-rw-r--r-- 1 root root 31911 Jul  3 14:56 sun50i-h6-beelink-gs1.dtb
-rw-r--r-- 1 root root 33042 Jul  3 14:56 sun50i-h6-orangepi-3.dtb
-rw-r--r-- 1 root root 30504 Jul  3 14:56 sun50i-h6-orangepi-lite2.dtb
-rw-r--r-- 1 root root 30287 Jul  3 14:56 sun50i-h6-orangepi-one-plus.dtb
-rw-r--r-- 1 root root 32368 Jul  3 14:56 sun50i-h6-pine-h64.dtb
-rw-r--r-- 1 root root 32882 Jul  3 14:56 sun50i-h6-pine-h64-model-b.dtb
-rw-r--r-- 1 root root 29544 Jul  3 14:56 sun50i-h6-tanix-tx6.dtb
-rw-r--r-- 1 root root 29305 Jul  3 14:56 sun50i-h6-tanix-tx6-mini.dtb

[manjaro@manjaro-arm ~]$ cat /boot/boot.txt
#
# /boot/boot.txt
# After modifying, run "pp-uboot-mkscr" to re-generate the U-Boot boot script.
#

#
# This is the description of the GPIO lines used in this boot script:
#
# GPIO #98 is PD2, or A64 ball W19, which controls the vibrator motor
# GPIO #114 is PD18, or A64 ball AB13, which controls the red part of the multicolor LED
# GPIO #115 is PD19, or A64 ball AB12, which controls the green part of the multicolor LED
# GPIO #116 is PD20, or A64 ball AB11, which controls the blue part of the multicolor LED
#

gpio set 98
gpio set 114

# Set root partition to the second partition of boot device
part uuid ${devtype} ${devnum}:1 uuid_boot
part uuid ${devtype} ${devnum}:2 uuid_root

setenv bootargs loglevel=4 console=tty0 console=${console} earlycon=uart,mmio32,0x01c28000 consoleblank=0 boot=PARTUUID=${uuid_boot} root=PARTUUID=${uuid_root} rw rootwait quiet audit=0 bootsplash.bootfile=bootsplash-themes/manjaro/bootsplash

if load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} /Image; then
  gpio clear 98
  if load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} /dtbs/${fdtfile}; then
    if load ${devtype} ${devnum}:${distro_bootpart} ${ramdisk_addr_r} /initramfs-linux.img; then
      gpio set 115
      booti ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r};
    else
      gpio set 116
      booti ${kernel_addr_r} - ${fdt_addr_r};
    fi;
  fi;
fi

# EOF