lupyuen/lupyuen.org
1
0
Fork 0
mirror of https://github.com/lupyuen/lupyuen.github.io.git synced 2025-01-13 10:18:33 +08:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
lupyuen.org/articles/interrupt.html

628 lines
No EOL
37 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="rustdoc">
<title>NuttX RTOS for PinePhone: Fixing the Interrupts</title>
<!-- Begin scripts/articles/*-header.html: Article Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<meta property="og:title"
content="NuttX RTOS for PinePhone: Fixing the Interrupts"
data-rh="true">
<meta property="og:description"
content="How Pine64 PinePhone handles Arm64 Interrupts with the Generic Interrupt Controller... And how we implemented PinePhone Interrupt Handling in Apache NuttX RTOS"
data-rh="true">
<meta name="description"
content="How Pine64 PinePhone handles Arm64 Interrupts with the Generic Interrupt Controller... And how we implemented PinePhone Interrupt Handling in Apache NuttX RTOS">
<meta property="og:image"
content="https://lupyuen.github.io/images/interrupt-title.jpg">
<meta property="og:type"
content="article" data-rh="true">
<link rel="canonical" href="https://lupyuen.org/articles/interrupt.html" />
<!-- End scripts/articles/*-header.html -->
<!-- Begin scripts/rustdoc-header.html: Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<link rel="alternate" type="application/rss+xml" title="RSS Feed for lupyuen" href="/rss.xml" />
<link rel="stylesheet" type="text/css" href="../normalize.css">
<link rel="stylesheet" type="text/css" href="../rustdoc.css" id="mainThemeStyle">
<link rel="stylesheet" type="text/css" href="../dark.css">
<link rel="stylesheet" type="text/css" href="../light.css" id="themeStyle">
<link rel="stylesheet" type="text/css" href="../prism.css">
<script src="../storage.js"></script><noscript>
<link rel="stylesheet" href="../noscript.css"></noscript>
<link rel="shortcut icon" href="../favicon.ico">
<style type="text/css">
#crate-search {
background-image: url("../down-arrow.svg");
}
</style>
<!-- End scripts/rustdoc-header.html -->
</head>
<body class="rustdoc">
<!--[if lte IE 8]>
<div class="warning">
This old browser is unsupported and will most likely display funky
things.
</div>
<![endif]-->
<!-- Begin scripts/rustdoc-before.html: Pre-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker -->
<div class="theme-picker" style="left: 0"><button id="theme-picker" aria-label="Pick another theme!"><img src="../brush.svg"
width="18" alt="Pick another theme!"></button>
<div id="theme-choices"></div>
</div>
<!-- Theme Picker -->
<!-- End scripts/rustdoc-before.html -->
<h1 class="title">NuttX RTOS for PinePhone: Fixing the Interrupts</h1>
<nav id="rustdoc"><ul>
<li><a href="#generic-interrupt-controller" title="Generic Interrupt Controller">1 Generic Interrupt Controller</a><ul></ul></li>
<li><a href="#allwinner-a64-gic" title="Allwinner A64 GIC">2 Allwinner A64 GIC</a><ul></ul></li>
<li><a href="#gic-version-2" title="GIC Version 2">3 GIC Version 2</a><ul></ul></li>
<li><a href="#test-pinephone-gic-with-qemu" title="Test PinePhone GIC with QEMU">4 Test PinePhone GIC with QEMU</a><ul></ul></li>
<li><a href="#pinephone-hangs-at-startup" title="PinePhone Hangs At Startup">5 PinePhone Hangs At Startup</a><ul></ul></li>
<li><a href="#timer-interrupt-isnt-handled" title="Timer Interrupt Isn’t Handled">6 Timer Interrupt Isn’t Handled</a><ul></ul></li>
<li><a href="#arm64-vector-table-is-wrong" title="Arm64 Vector Table Is Wrong">7 Arm64 Vector Table Is Wrong</a><ul></ul></li>
<li><a href="#exception-levels" title="Exception Levels">8 Exception Levels</a><ul></ul></li>
<li><a href="#whats-next" title="What’s Next">9 What’s Next</a><ul></ul></li></ul></nav><p>📝 <em>1 Sep 2022</em></p>
<p><img src="https://lupyuen.github.io/images/interrupt-title.jpg" alt="Tracing Arm64 Interrupts on QEMU Emulator can get… Really messy" /></p>
<p><strong>UPDATE:</strong> PinePhone is now officially supported by Apache NuttX RTOS <a href="https://lupyuen.github.io/articles/what">(See this)</a></p>
<p>Creating our own <strong>Operating System</strong> (non-Linux) for <a href="https://wiki.pine64.org/index.php/PinePhone"><strong>Pine64 PinePhone</strong></a> can be super challenging…</p>
<ul>
<li>
<p>How does PinePhone handle Interrupts?</p>
</li>
<li>
<p>What’s a Generic Interrupt Controller? (GIC)</p>
</li>
<li>
<p>Why is PinePhone’s GIC particularly problematic?</p>
</li>
<li>
<p>What’s an Exception Level? (EL)</p>
</li>
<li>
<p>Why does EL matter for handling Arm64 Interrupts?</p>
</li>
</ul>
<p>We’ll answer these questions today as we port <a href="https://lupyuen.github.io/articles/uboot"><strong>Apache NuttX RTOS</strong></a> to PinePhone.</p>
<p>Let’s dive into our <strong>Porting Journal</strong> for NuttX on PinePhone…</p>
<ul>
<li><a href="https://github.com/lupyuen/pinephone-nuttx"><strong>lupyuen/pinephone-nuttx</strong></a></li>
</ul>
<p>And relive the very first <strong>Interrupt issue</strong> that we hit…</p>
<div class="example-wrap"><pre class="language-text"><code>HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
arm64_gic_initialize: no distributor detected, giving up</code></pre></div>
<p><img src="https://lupyuen.github.io/images/interrupt-peripheral.jpg" alt="Partial list of Shared Peripheral Interrupts for Allwinner A64’s GIC" /></p>
<p><em>Partial list of Shared Peripheral Interrupts for Allwinner A64’s GIC</em></p>
<h1 id="generic-interrupt-controller"><a class="doc-anchor" href="#generic-interrupt-controller">§</a>1 Generic Interrupt Controller</h1>
<p><em>What’s a GIC?</em></p>
<p>PinePhone’s <strong>Generic Interrupt Controller (GIC)</strong> works like a typical Interrupt Controller in a CPU. It manages Interrupts for the Arm64 CPU.</p>
<p>Except that GIC is a special chunk of silicon that lives <strong>inside the Allwinner A64 SoC</strong>. (Outside the Arm64 CPU)</p>
<p><em>Huh? Arm64 CPU doesn’t have its own Interrupt Controller?</em></p>
<p>Interrupting gets complicated… Remember PinePhone runs on <strong>4 Arm64 CPUs?</strong></p>
<p>The 4 CPUs must handle the Interrupts triggered by <strong>all kinds of Peripherals</strong>: UART, I2C, SPI, DMA, USB, microSD, eMMC, …</p>
<p>We do this the <strong>flexible, efficient</strong> way with a GIC, which supports…</p>
<ul>
<li>
<p><strong>Shared Peripheral Interrupts (SPI)</strong></p>
<p>GIC can route Peripheral Interrupts to <strong>one or multiple CPUs</strong></p>
<p>(Pic above)</p>
</li>
<li>
<p><strong>Private Peripheral Interrupts (PPI)</strong></p>
<p>GIC can route Peripheral Interrupts to a <strong>single CPU</strong></p>
</li>
<li>
<p><strong>Software-Generated Interrupts (SGI)</strong></p>
<p>GIC lets CPUs to <strong>talk to each other</strong> by triggering Software Interrupts</p>
<p>(Anyone remember Silicon Graphics?)</p>
</li>
</ul>
<p>Allwinner A64’s GIC supports <strong>157 Interrupt Sources</strong>: 16 Software-Generated, 16 Private and 125 Shared.</p>
<p>The GIC in Allwinner A64 is a little problematic, let’s talk…</p>
<p><img src="https://lupyuen.github.io/images/interrupt-gic.jpg" alt="Allwinner A64 runs on Arm GIC Version 2" /></p>
<p><em>Allwinner A64 runs on Arm GIC Version 2</em></p>
<h1 id="allwinner-a64-gic"><a class="doc-anchor" href="#allwinner-a64-gic">§</a>2 Allwinner A64 GIC</h1>
<p><em>What’s this GIC error we saw earlier?</em></p>
<div class="example-wrap"><pre class="language-text"><code>- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
arm64_gic_initialize: no distributor detected, giving up</code></pre></div>
<p>When we boot NuttX RTOS, it expects PinePhone to provide a modern <strong>Generic Interrupt Controller (GIC), Version 3</strong>.</p>
<p>But the <a href="https://dl.linux-sunxi.org/A64/A64_Datasheet_V1.1.pdf"><strong>Allwinner A64 User Manual</strong></a> (page 210, “GIC”) says that PinePhone runs on…</p>
<ul>
<li>
<p><a href="https://developer.arm.com/documentation/ddi0471/b/introduction/about-the-gic-400"><strong>Arm GIC PL400</strong></a>, which is based on…</p>
</li>
<li>
<p><a href="https://developer.arm.com/documentation/ihi0048/latest/"><strong>Arm GIC Version 2</strong></a></p>
</li>
</ul>
<p>Our GIC Version 2 is from 2011, when Arm CPUs were still 32-bit… That’s <strong>11 years ago!</strong></p>
<p>So we need to fix NuttX and downgrade GIC Version 3 <strong>back to GIC Version 2</strong>, specially for PinePhone.</p>
<p><em>We’re sure that PinePhone runs on GIC Version 2?</em></p>
<p>Let’s verify! This code reads the <strong>GIC Version</strong> from PinePhone: <a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L710-L734">arch/arm64/src/common/arm64_gicv3.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Init GIC v2 for PinePhone
int arm64_gic_initialize(void) {
sinfo(&quot;TODO: Init GIC for PinePhone\n&quot;);
sinfo(&quot;CONFIG_GICD_BASE=%p\n&quot;, CONFIG_GICD_BASE);
sinfo(&quot;CONFIG_GICR_BASE=%p\n&quot;, CONFIG_GICR_BASE);
// To verify the GIC Version, read the Peripheral ID2 Register (ICPIDR2) at Offset 0xFE8 of GIC Distributor.
// Bits 4 to 7 of ICPIDR2 are...
// - 0x1 for GIC Version 1
// - 0x2 for GIC Version 2
// GIC Distributor is at 0x01C80000 + 0x1000
const uint8_t *ICPIDR2 = (const uint8_t *) (CONFIG_GICD_BASE + 0xFE8);
uint8_t version = (*ICPIDR2 &gt;&gt; 4) &amp; 0b1111;
sinfo(&quot;GIC Version is %d\n&quot;, version);
DEBUGASSERT(version == 2);</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/arm64/src/common/arm64_gicv2.c#L669-L706">(Update)</a></p>
<p>Here’s the output…</p>
<div class="example-wrap"><pre class="language-text"><code>TODO: Init GIC for PinePhone
CONFIG_GICD_BASE=0x1c81000
CONFIG_GICR_BASE=0x1c82000
GIC Version is 2</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx#pinephone-u-boot-log">(Source)</a></p>
<p>Yep PinePhone runs on <strong>GIC Version 2</strong>. Bummer.</p>
<p><em>What are GICD and GICR?</em></p>
<p><strong>GICD (GIC Distributor)</strong> and <strong>GICR (GIC CPU Interface)</strong> are the addresses for accessing the GIC on PinePhone.</p>
<p>According to <a href="https://dl.linux-sunxi.org/A64/A64_Datasheet_V1.1.pdf"><strong>Allwinner A64 User Manual</strong></a> (page 74, “Memory Mapping”), the GIC is located at…</p>
<div><table><thead><tr><th style="text-align: left">Module</th><th style="text-align: left">Address</th><th style="text-align: left">Remarks</th></tr></thead><tbody>
<tr><td style="text-align: left">GIC_DIST</td><td style="text-align: left"><code>0x01C8</code> <code>0000</code> + <code>0x1000</code></td><td style="text-align: left">GIC Distributor (GICD)</td></tr>
<tr><td style="text-align: left">GIC_CPUIF</td><td style="text-align: left"><code>0x01C8</code> <code>0000</code> + <code>0x2000</code></td><td style="text-align: left">GIC CPU Interface (GICR)</td></tr>
</tbody></table>
</div>
<p>Which we define in NuttX as: <a href="https://github.com/apache/nuttx/blob/master/arch/arm64/include/a64/chip.h#L39-L44">arch/arm64/include/a64/chip.h</a></p>
<div class="example-wrap"><pre class="language-c"><code>// PinePhone Generic Interrupt Controller
// GIC_DIST: 0x01C80000 + 0x1000
// GIC_CPUIF: 0x01C80000 + 0x2000
#define CONFIG_GICD_BASE 0x01C81000
#define CONFIG_GICR_BASE 0x01C82000 </code></pre></div>
<p>Back to our headache of GIC Version 2…</p>
<h1 id="gic-version-2"><a class="doc-anchor" href="#gic-version-2">§</a>3 GIC Version 2</h1>
<p><em>Does NuttX support GIC Version 2 for PinePhone?</em></p>
<p>Yes NuttX supports <strong>Generic Interrupt Controller (GIC) Version 2</strong> but there’s a catch… It’s for <strong>Arm32 CPUs, not Arm64 CPUs!</strong></p>
<ul>
<li>
<p><a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm/src/armv7-a/arm_gicv2.c">arch/arm/src/armv7-a/arm_gicv2.c</a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm/src/armv7-a/arm_gicv2_dump.c">arch/arm/src/armv7-a/arm_gicv2_dump.c</a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm/src/armv7-a/gic.h">arch/arm/src/armv7-a/gic.h</a></p>
</li>
</ul>
<p>Remember: GIC Version 2 was created for Arm32.</p>
<p><em>So we port NuttX’s GIC Version 2 from Arm32 to Arm64?</em></p>
<p>Kinda. We did a <strong>horrible hack</strong>… Don’t try this at home! (Unless you have a ten-foot pole) <a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L765-L823">arch/arm64/src/common/arm64_gicv3.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// GIC v2 for PinePhone:
// Reuse the implementation of Arm32 GIC v2
#define PINEPHONE_GICv2
#define CONFIG_ARMV7A_HAVE_GICv2
#define CONFIG_ARCH_TRUSTZONE_NONSECURE
// Override...
// MPCORE_ICD_VBASE: GIC Distributor
// MPCORE_ICC_VBASE: GIC CPU Interface
#include &quot;../arch/arm/src/armv7-a/mpcore.h&quot;
#undef MPCORE_ICD_VBASE
#undef MPCORE_ICC_VBASE
#define MPCORE_ICD_VBASE CONFIG_GICD_BASE // 0x01C81000
#define MPCORE_ICC_VBASE CONFIG_GICR_BASE // 0x01C82000
// Inject Arm32 GIC v2 Implementation
#include &quot;../arch/arm/src/armv7-a/arm_gicv2.c&quot;</code></pre></div>
<p><a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c">(We commented out the <strong>GIC Version 3</strong> code as <strong><code>NOTUSED</code></strong>)</a></p>
<p><em>What! Did we just <code>#include</code> the GIC Version 2 Source Code from Arm32 into Arm64?</em></p>
<p>Yep it’s an awful trick but it seems to work!</p>
<p>We made <strong>minor tweaks</strong> to GIC Version 2 to compile with Arm64…</p>
<ul>
<li>
<p><a href="https://github.com/lupyuen/nuttx/commit/6fa0e7e5d2beddad07890c83d2ee428a3f2b8a62#diff-6e1132aef124dabaf94c200ab06d65c7bc2b9967bf76a46aba71a7f43b5fb219">Changes to arm_gicv2.c</a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx/commit/4fc2669fef62d12ba1dd428f2daf03d3bc362501#diff-eb05c977988d59202a9472f6fa7f9dc290724662ad6d15a4ba99b8f1fc1dc8f8">Changes to arm_gicv2_dump.c</a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx/commit/6fa0e7e5d2beddad07890c83d2ee428a3f2b8a62#diff-b4fcb67b71de954c942ead9bb0868e720a5802c90743f0a1883f84b7565e1a0f">Changes to gic.h</a></p>
</li>
</ul>
<p>We rewrote this function for Arm64 because we’re passing <strong>64-bit Registers</strong> (instead of 32-bit): <a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L795-L822">arm64_gicv3.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Decode IRQ for PinePhone.
// Based on arm_decodeirq in arm_gicv2.c.
// Previously we passed 32-bit Registers as `uint32_t *`
uint64_t * arm64_decodeirq(uint64_t * regs) {
/* Omitted: Get the interrupt ID */
...
/* Dispatch the Arm64 interrupt */
regs = arm64_doirq(irq, regs);</code></pre></div>
<p>Everything else stays the same! Well except for…</p>
<ul>
<li>
<p><a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L713-L743"><strong><code>arm64_gic_initialize</code></strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L753-L760"><strong><code>arm64_gic_secondary_init</code></strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L162-L196"><strong><code>arm64_gic_irq_set_priority</code></strong></a></p>
</li>
</ul>
<p><em>Injecting Arm32 code into Arm64 sounds so reckless… Will it work?</em></p>
<p>Let’s test our reckless GIC Version 2 with QEMU Emulator…</p>
<p><strong>UPDATE:</strong> NuttX Mainline now supports GIC Version 2 <a href="https://github.com/apache/nuttx/blob/master/arch/arm64/src/common/arm64_gicv2.c">(See this)</a></p>
<p><img src="https://lupyuen.github.io/images/interrupt-title2.png" alt="Tracing Arm64 Interrupts on QEMU Emulator can get… Really messy" /></p>
<p><a href="https://github.com/lupyuen/nuttx/blob/3331de1b84fd7579edfe726abb71b18beeac29e6/nuttx.log"><em>Tracing Arm64 Interrupts on QEMU Emulator can get… Really messy</em></a></p>
<h1 id="test-pinephone-gic-with-qemu"><a class="doc-anchor" href="#test-pinephone-gic-with-qemu">§</a>4 Test PinePhone GIC with QEMU</h1>
<p><em>Will our hacked GIC Version 2 run on PinePhone?</em></p>
<p>Before testing on PinePhone, let’s test our Generic Interrupt Controller (GIC) Version 2 on <a href="https://www.qemu.org/"><strong>QEMU Emulator</strong></a>.</p>
<p>Follow these steps to build NuttX for <strong>QEMU with GIC Version 2</strong>…</p>
<ul>
<li><a href="https://github.com/lupyuen/pinephone-nuttx#test-pinephone-gic-with-qemu"><strong>“Test PinePhone GIC with QEMU”</strong></a></li>
</ul>
<p>Enter this to <strong>start QEMU with NuttX</strong> and GIC Version 2…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Run GIC Version 2 with QEMU
qemu-system-aarch64 \
-smp 4 \
-cpu cortex-a53 \
-nographic \
-machine virt,virtualization=on,gic-version=2 \
-net none \
-chardev stdio,id=con,mux=on \
-serial chardev:con \
-mon chardev=con,mode=readline \
-kernel ./nuttx</code></pre></div>
<p>Note that “<strong><code>gic-version=2</code></strong>” instead of the usual GIC Version 3 for Arm64.</p>
<p>Also we simulated 4 Cores of Arm Cortex-A53 (similar to PinePhone): “<strong><code>-smp 4</code></strong>”</p>
<p>We see this in QEMU…</p>
<div class="example-wrap"><pre class="language-text"><code>- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x402c4000, heap_size=0x7d3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x8000000
arm64_gic_initialize: CONFIG_GICR_BASE=0x8010000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 62.50MHz, cycle 62500
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x402a7000 _einit: 0x402a7000 _stext: 0x40280000 _etext: 0x402a8000
nsh: sysinit: fopen failed: 2
nsh: mkfatfs: command not found
NuttShell (NSH) NuttX-10.3.0-RC2
nsh&gt;
nx_start: CPU0: Beginning Idle Loop</code></pre></div>
<p><a href="https://github.com/lupyuen/nuttx/blob/3331de1b84fd7579edfe726abb71b18beeac29e6/nuttx.log">(See the Complete Log)</a></p>
<p>NuttX with GIC Version 2 boots OK on QEMU, and will probably run on PinePhone!</p>
<p><em>We tested Interrupts with GIC Version 2?</em></p>
<p>Yep the pic above shows <strong>“TX”</strong> whenever an Interrupt Handler is dispatched.</p>
<p>(We added Debug Logging to <a href="https://github.com/lupyuen/nuttx/blob/gicv2/arch/arm64/src/common/arm64_vectors.S#L337-L350">arm64_vectors.S</a> and <a href="https://github.com/lupyuen/nuttx/blob/gicv2/arch/arm64/src/common/arm64_vector_table.S#L47-L75">arm64_vector_table.S</a>)</p>
<p><em>How did we get the GIC Base Addresses for QEMU?</em></p>
<div class="example-wrap"><pre class="language-text"><code>CONFIG_GICD_BASE=0x8000000
CONFIG_GICR_BASE=0x8010000</code></pre></div>
<p>We got the Base Addresses for GIC Distributor (<strong><code>CONFIG_GICD_BASE</code></strong>) and GIC CPU Interface (<strong><code>CONFIG_GICR_BASE</code></strong>) by dumping the Device Tree from QEMU…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Dump Device Tree for GIC Version 2
qemu-system-aarch64 \
-smp 4 \
-cpu cortex-a53 \
-nographic \
-machine virt,virtualization=on,gic-version=2,dumpdtb=gicv2.dtb \
-net none \
-chardev stdio,id=con,mux=on \
-serial chardev:con \
-mon chardev=con,mode=readline \
-kernel ./nuttx
## Convert Device Tree to text format
dtc \
-o gicv2.dts \
-O dts \
-I dtb \
gicv2.dtb</code></pre></div>
<p>The Base Addresses are revealed in the <strong>GIC Version 2 Device Tree</strong>: <a href="https://github.com/lupyuen/nuttx/blob/gicv2/gicv2.dts#L324">gicv2.dts</a>…</p>
<div class="example-wrap"><pre class="language-text"><code>intc@8000000 {
reg = &lt;
0x00 0x8000000 0x00 0x10000 // GIC Distributor: 0x8000000
0x00 0x8010000 0x00 0x10000 // GIC CPU Interface: 0x8010000
0x00 0x8030000 0x00 0x10000 // VGIC Virtual Interface Control: 0x8030000
0x00 0x8040000 0x00 0x10000 // VGIC Virtual CPU Interface: 0x8040000
&gt;;
compatible = &quot;arm,cortex-a15-gic&quot;;</code></pre></div>
<p><a href="https://www.kernel.org/doc/Documentation/devicetree/bindings/interrupt-controller/arm%2Cgic.txt">(More about this)</a></p>
<p>Which we defined in NuttX at…</p>
<ul>
<li><a href="https://github.com/apache/nuttx/blob/master/arch/arm64/include/qemu/chip.h#L37-L51">arch/arm64/include/qemu/chip.h</a></li>
</ul>
<p><strong>UPDATE:</strong> NuttX Mainline now provides a Board Config “<code>qemu-armv8a:nsh_gicv2</code>” for testing GIC Version 2 with QEMU <a href="qemu-armv8a:nsh_gicv2">(See this)</a></p>
<h1 id="pinephone-hangs-at-startup"><a class="doc-anchor" href="#pinephone-hangs-at-startup">§</a>5 PinePhone Hangs At Startup</h1>
<p><em>NuttX should boot OK on PinePhone right?</em></p>
<p>We followed these steps to <strong>boot NuttX on PinePhone</strong> (with GIC Version 2)…</p>
<ul>
<li><a href="https://github.com/lupyuen/pinephone-nuttx#nuttx-boot-log"><strong>“NuttX Boot Log”</strong></a></li>
</ul>
<p>But <strong>NuttX got stuck</strong> on PinePhone in a very curious way…</p>
<div class="example-wrap"><pre class="language-text"><code>arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
uart_regi</code></pre></div>
<p>NuttX got stuck <strong>while printing a line!</strong></p>
<p>And it happened a short while after we started the <strong>System Timer</strong>: <strong><code>up_timer_initialize</code></strong></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx#system-timer">(More about System Timer)</a></p>
<p><em>Something in the System Timer caused this?</em></p>
<p>Yep! If we <strong>disabled the System Timer</strong>, PinePhone will continue to boot.</p>
<p>Remember that the System Timer will trigger Interrupts periodically…</p>
<p>Perhaps we’re <strong>handling Interrupts incorrectly?</strong></p>
<p>Let’s investigate…</p>
<p><strong>UPDATE:</strong> This problem doesn’t happen with the latest code in NuttX Mainline <a href="https://lupyuen.github.io/articles/uboot#porting-notes">(See this)</a></p>
<h1 id="timer-interrupt-isnt-handled"><a class="doc-anchor" href="#timer-interrupt-isnt-handled">§</a>6 Timer Interrupt Isn’t Handled</h1>
<p><em>Why did PinePhone hang while handling System Timer Interrupts?</em></p>
<p><em>Was the Timer Interrupt Handler called?</em></p>
<p>We verified that <strong>Timer Interrupt Handler <a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.c#L134-L161">arm64_arch_timer_compare_isr</a></strong> was NEVER called.</p>
<p>(We checked by calling <a href="https://github.com/lupyuen/pinephone-nuttx#boot-debugging"><strong><code>up_putc</code></strong></a>, which prints directly to the UART Port)</p>
<p>So something went wrong BEFORE calling the Interrupt Handler. Let’s backtrack…</p>
<p><em>Is the Interrupt Vector Table pointing correctly to the Timer Interrupt Handler?</em></p>
<p>NuttX defines an <strong>Interrupt Vector Table</strong> for dispatching Interrupt Handlers…</p>
<ul>
<li><a href="https://github.com/lupyuen/pinephone-nuttx#handling-interrupts"><strong>“Handling Interrupts”</strong></a></li>
</ul>
<p>We dumped NuttX’s Interrupt Vector Table…</p>
<ul>
<li><a href="https://github.com/lupyuen/pinephone-nuttx#dump-interrupt-vector-table"><strong>“Dump Interrupt Vector Table”</strong></a></li>
</ul>
<p>And verified that the Timer Interrupt Handler is set correctly in the table.</p>
<p><em>Maybe something went wrong when NuttX tried to call the Interrupt Handler?</em></p>
<p>NuttX should call <a href="https://github.com/lupyuen/pinephone-nuttx#handling-interrupts"><strong>Interrupt Dispatcher <code>irq_dispatch</code></strong></a> to dispatch the Interrupt Handler…</p>
<p>But nope, <strong><code>irq_dispatch</code></strong> was never called.</p>
<p><em>Some error occurred and NuttX threw an Unexpected Interrupt?</em></p>
<p>Nope, the <a href="https://github.com/lupyuen/pinephone-nuttx#handling-interrupts"><strong>Unexpected Interrupt Handler <code>irq_unexpected_isr</code></strong></a> was never called either.</p>
<p><em>OK I’m really stumped. Did something go bad deep inside Arm64 Interrupts?</em></p>
<p>Possibly! Let’s talk about the Arm64 Vector Table…</p>
<p><img src="https://lupyuen.github.io/images/interrupt-vbar.jpg" alt="Vector Base Address Register, EL1" /></p>
<h1 id="arm64-vector-table-is-wrong"><a class="doc-anchor" href="#arm64-vector-table-is-wrong">§</a>7 Arm64 Vector Table Is Wrong</h1>
<p><em>When an Interrupt is triggered, what happens in the Arm64 CPU?</em></p>
<p>According to the <a href="https://documentation-service.arm.com/static/5e9075f9c8052b1608761519?token="><strong>Arm Cortex-A53 Technical Reference Manual</strong></a> (page 4-121), the CPU reads the <strong>Vector Base Address Register (EL1)</strong> to locate the Arm64 Vector Table. (Pic above)</p>
<p>(Why EL1? We’ll explain in a while)</p>
<p>The <strong>Arm64 Vector Table</strong> looks like this…</p>
<p><img src="https://lupyuen.github.io/images/interrupt-vector.png" alt="Arm64 Vector Table" /></p>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/arm64/src/common/arm64_vector_table.S#L92-L130">(Source)</a></p>
<p>Which we define in NuttX as <strong><code>_vector_table</code></strong>: <a href="https://github.com/apache/nuttx/blob/master/arch/arm64/src/common/arm64_vector_table.S#L130-L233">arch/arm64/src/common/arm64_vector_table.S</a></p>
<div class="example-wrap"><pre class="language-text"><code>GTEXT(_vector_table)
SECTION_SUBSEC_FUNC(exc_vector_table,_vector_table_section,_vector_table)
...
/* Current EL with SP0 / IRQ */
.align 7
arm64_enter_exception x0, x1
b arm64_irq_handler
...
/* Current EL with SPx / IRQ */
.align 7
arm64_enter_exception x0, x1
b arm64_irq_handler</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/arm64/src/common/arm64_vector_table.S#L41-L87">(<strong><code>arm64_enter_exception</code></strong> saves the Arm64 Registers)</a></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx#handling-interrupts">(<strong><code>arm64_irq_handler</code></strong> is the NuttX IRQ Handler)</a></p>
<p><em>So Vector Base Address Register (EL1) should point to <code>_vector_table</code>?</em></p>
<p>Let’s find out! This is how we read <strong>Vector Base Address Register (EL1)</strong>: <a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.c#L212-L235">arch/arm64/src/common/arm64_arch_timer.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>void up_timer_initialize(void) {
...
// Read Vector Base Address Register EL1
extern void *_vector_table[];
sinfo(&quot;_vector_table=%p\n&quot;, _vector_table);
sinfo(&quot;Before writing: vbar_el1=%p\n&quot;, read_sysreg(vbar_el1));</code></pre></div>
<p>Here’s the output on PinePhone…</p>
<div class="example-wrap"><pre class="language-text"><code>_vector_table=0x400a7000
Before writing: vbar_el1=0x40227000</code></pre></div>
<p>Aha! <strong><code>_vector_table</code></strong> is at <strong><code>0x400a</code> <code>7000</code></strong>… But Vector Base Address Register (EL1) says <strong><code>0x4022</code> <code>7000</code>!</strong></p>
<p>Our Arm64 CPU is pointing to the <strong>wrong Arm64 Vector Table</strong>… Hence our Interrupt Handler is never called!</p>
<p>Let’s fix it: <a href="https://github.com/lupyuen/nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.c#L212-L235">arch/arm64/src/common/arm64_arch_timer.c</a></p>
<div class="example-wrap"><pre class="language-c"><code> // Write Vector Base Address Register EL1
write_sysreg((uint64_t)_vector_table, vbar_el1);
ARM64_ISB();
// Read Vector Base Address Register EL1
sinfo(&quot;After writing: vbar_el1=%p\n&quot;, read_sysreg(vbar_el1));</code></pre></div>
<p>This writes the correct value of <strong><code>_vector_table</code></strong> back into Vector Base Address Register EL1. Here’s the output on PinePhone…</p>
<div class="example-wrap"><pre class="language-text"><code>_vector_table=0x400a7000
Before writing: vbar_el1=0x40227000
After writing: vbar_el1=0x400a7000</code></pre></div>
<p>Yep Vector Base Address Register (EL1) is now correct.</p>
<p>Our Interrupt Handlers are now working fine… And PinePhone boots successfully yay! 🎉</p>
<div class="example-wrap"><pre class="language-text"><code>Starting kernel ...
HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x400c4000, heap_size=0x7f3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
up_timer_initialize: _vector_table=0x400a7000
up_timer_initialize: Before writing: vbar_el1=0x40227000
up_timer_initialize: After writing: vbar_el1=0x400a7000
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x400a7000 _einit: 0x400a7000 _stext: 0x40080000 _etext: 0x400a8000
nsh: sysinit: fopen failed: 2
nshn:x _msktfaarttf:s :C PcUo0m:m aBnedg innonti nfgo uInddle L oNouptt
Shell (NSH) NuttX-10.3.0-RC2</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx#garbled-console-output">(Yeah the output is slightly garbled, here’s the workaround)</a></p>
<p>Now that we have UART Interrupts, <strong>NuttX Shell</strong> works perfectly OK on PinePhone…</p>
<div class="example-wrap"><pre class="language-text"><code>nsh&gt; uname -a
NuttX 10.3.0-RC2 fc909c6-dirty Sep 1 2022 17:05:44 arm64 qemu-armv8a
nsh&gt; help
help usage: help [-v] [&lt;cmd&gt;]
. cd dmesg help mount rmdir true xd
[ cp echo hexdump mv set truncate
? cmp exec kill printf sleep uname
basename dirname exit ls ps source umount
break dd false mkdir pwd test unset
cat df free mkrd rm time usleep
Builtin Apps:
getprime hello nsh ostest sh
nsh&gt; hello
task_spawn: name=hello entry=0x4009b1a0 file_actions=0x400c9580 attr=0x400c9588 argv=0x400c96d0
spawn_execattrs: Setting policy=2 priority=100 for pid=3
Hello, World!!
nsh&gt; ls /dev
/dev:
console
null
ram0
ram2
ttyS0
zero</code></pre></div>
<p><a href="https://youtube.com/shorts/WmRzfCiWV6o?feature=share"><strong>Watch the Demo on YouTube</strong></a></p>
<p><a href="https://youtu.be/MJDxCcKAv0g"><strong>Another Demo Video</strong></a></p>
<p>Let’s talk about EL1…</p>
<p><strong>UPDATE:</strong> This patching isn’t needed with the latest code in NuttX Mainline <a href="https://lupyuen.github.io/articles/uboot#porting-notes">(See this)</a></p>
<h1 id="exception-levels"><a class="doc-anchor" href="#exception-levels">§</a>8 Exception Levels</h1>
<p><em>What’s EL1?</em></p>
<p>EL1 is <strong>Exception Level 1</strong>. As defined in <a href="https://documentation-service.arm.com/static/5e9075f9c8052b1608761519?token="><strong>Arm Cortex-A53 Technical Reference Manual</strong></a> page 3-5 (“Exception Level”)…</p>
<blockquote>
<p>The ARMv8 exception model defines exception levels EL0-EL3, where:</p>
</blockquote>
<blockquote>
<ul>
<li>EL0 has the lowest software execution privilege, and execution at EL0 is called unprivileged execution.</li>
</ul>
</blockquote>
<blockquote>
<ul>
<li>Increased exception levels, from 1 to 3, indicate increased software execution privilege.</li>
</ul>
</blockquote>
<blockquote>
<ul>
<li>EL2 provides support for processor virtualization.</li>
</ul>
</blockquote>
<blockquote>
<ul>
<li>EL3 provides support for a secure state, see Security state on page 3-6.</li>
</ul>
</blockquote>
<p>So <strong>EL1 is (kinda) privileged</strong>, suitable for running OS Kernel code. (Like NuttX)</p>
<p>NuttX runs mostly in <strong>EL1</strong> and briefly in <strong>EL2</strong> (at startup)…</p>
<div class="example-wrap"><pre class="language-text"><code>HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize</code></pre></div>
<p>(Remember that EL1 is less privileged than EL2, which supports Processor Virtualization. Host OS will run at EL2, Guest OS at EL1)</p>
<p>That’s why we talked about the EL1 Vector Base Address Register in the previous section.</p>
<p><em>So there’s a Vector Base Address Register for EL1, EL2 and EL3?</em></p>
<p>Indeed! Each Exception Level has its own Arm64 Vector Table.</p>
<p>(Except EL0)</p>
<p><em>Who loads the EL1 Vector Base Address Register?</em></p>
<p>The EL1 Vector Base Address Register is loaded during <strong>EL1 Initialisation</strong> at startup: <a href="https://github.com/apache/nuttx/blob/master/arch/arm64/src/common/arm64_boot.c#L132-L162">arch/arm64/src/common/arm64_boot.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>void arm64_boot_el1_init(void) {
/* Setup vector table */
write_sysreg((uint64_t)_vector_table, vbar_el1);
ARM64_ISB();</code></pre></div>
<p><strong><code>arm64_boot_el1_init</code></strong> is called by our Startup Code: <a href="https://github.com/apache/nuttx/blob/master/arch/arm64/src/common/arm64_head.S#L210-L227">arch/arm64/src/common/arm64_head.S</a></p>
<div class="example-wrap"><pre class="language-text"><code> PRINT(switch_el1, &quot;- Boot from EL1\r\n&quot;)
/* EL1 init */
bl arm64_boot_el1_init
/* set SP_ELx and Enable SError interrupts */
msr SPSel, #1
msr DAIFClr, #(DAIFCLR_ABT_BIT)
isb
jump_to_c_entry:
PRINT(jump_to_c_entry, &quot;- Boot to C runtime for OS Initialize\r\n&quot;)
ret x25</code></pre></div>
<p>The <strong>Boot Sequence</strong> for NuttX RTOS is explained here…</p>
<ul>
<li><a href="https://github.com/lupyuen/pinephone-nuttx#boot-sequence"><strong>“Boot Sequence”</strong></a></li>
</ul>
<h1 id="whats-next"><a class="doc-anchor" href="#whats-next">§</a>9 What’s Next</h1>
<p>There’s plenty to be done for NuttX on PinePhone, please lemme know if you would like to join me 🙏</p>
<p>Please check out the other articles on NuttX for PinePhone…</p>
<ul>
<li><a href="https://github.com/lupyuen/pinephone-nuttx"><strong>“Apache NuttX RTOS for PinePhone”</strong></a></li>
</ul>
<p>Many Thanks to my <a href="https://lupyuen.github.io/articles/sponsor"><strong>GitHub Sponsors</strong></a> for supporting my work! This article wouldn’t have been possible without your support.</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/sponsor"><strong>Sponsor me a coffee</strong></a></p>
</li>
<li>
<p><a href="https://www.reddit.com/r/PINE64official/comments/x2v02z/nuttx_rtos_on_pinephone_fixing_the_interrupts/"><strong>Discuss this article on Reddit</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-ox64"><strong>My Current Project: “Apache NuttX RTOS for Ox64 BL808”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-star64"><strong>My Other Project: “NuttX for Star64 JH7110”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx"><strong>Older Project: “NuttX for PinePhone”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io"><strong>Check out my articles</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/rss.xml"><strong>RSS Feed</strong></a></p>
</li>
</ul>
<p><em>Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…</em></p>
<p><a href="https://github.com/lupyuen/lupyuen.github.io/blob/master/src/interrupt.md"><strong>lupyuen.github.io/src/interrupt.md</strong></a></p>
<!-- Begin scripts/rustdoc-after.html: Post-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker and Prism Theme -->
<script src="../theme.js"></script>
<script src="../prism.js"></script>
<!-- Theme Picker and Prism Theme -->
<!-- End scripts/rustdoc-after.html -->
</body>
</html>
Reference in a new issue View git blame Copy permalink