mirror of
https://github.com/lupyuen/lupyuen.github.io.git
synced 2025-01-13 09:08:30 +08:00
861 lines
No EOL
52 KiB
HTML
861 lines
No EOL
52 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<meta name="generator" content="rustdoc">
|
||
<title>Automated Testing with Ox64 BL808 Emulator (Apache NuttX RTOS)</title>
|
||
|
||
|
||
<!-- Begin scripts/articles/*-header.html: Article Header for Custom Markdown files processed by rustdoc, like chip8.md -->
|
||
<meta property="og:title"
|
||
content="Automated Testing with Ox64 BL808 Emulator (Apache NuttX RTOS)"
|
||
data-rh="true">
|
||
<meta property="og:description"
|
||
content="Every day we're auto-building Apache NuttX RTOS for Ox64 BL808 SBC... Can we test NuttX on Ox64 Emulator automatically after building? Let's find out!"
|
||
data-rh="true">
|
||
<meta name="description"
|
||
content="Every day we're auto-building Apache NuttX RTOS for Ox64 BL808 SBC... Can we test NuttX on Ox64 Emulator automatically after building? Let's find out!">
|
||
<meta property="og:image"
|
||
content="https://lupyuen.github.io/images/tinyemu3-title.jpg">
|
||
<meta property="og:type"
|
||
content="article" data-rh="true">
|
||
<link rel="canonical"
|
||
href="https://lupyuen.org/articles/tinyemu3.html" />
|
||
<!-- End scripts/articles/*-header.html -->
|
||
<!-- Begin scripts/rustdoc-header.html: Header for Custom Markdown files processed by rustdoc, like chip8.md -->
|
||
<link rel="alternate" type="application/rss+xml" title="RSS Feed for lupyuen" href="/rss.xml" />
|
||
<link rel="stylesheet" type="text/css" href="../normalize.css">
|
||
<link rel="stylesheet" type="text/css" href="../rustdoc.css" id="mainThemeStyle">
|
||
<link rel="stylesheet" type="text/css" href="../dark.css">
|
||
<link rel="stylesheet" type="text/css" href="../light.css" id="themeStyle">
|
||
<link rel="stylesheet" type="text/css" href="../prism.css">
|
||
<script src="../storage.js"></script><noscript>
|
||
<link rel="stylesheet" href="../noscript.css"></noscript>
|
||
<link rel="shortcut icon" href="../favicon.ico">
|
||
<style type="text/css">
|
||
#crate-search {
|
||
background-image: url("../down-arrow.svg");
|
||
}
|
||
</style>
|
||
<!-- End scripts/rustdoc-header.html -->
|
||
|
||
|
||
</head>
|
||
<body class="rustdoc">
|
||
<!--[if lte IE 8]>
|
||
<div class="warning">
|
||
This old browser is unsupported and will most likely display funky
|
||
things.
|
||
</div>
|
||
<![endif]-->
|
||
|
||
|
||
<!-- Begin scripts/rustdoc-before.html: Pre-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
|
||
|
||
<!-- Begin Theme Picker -->
|
||
<div class="theme-picker" style="left: 0"><button id="theme-picker" aria-label="Pick another theme!"><img src="../brush.svg"
|
||
width="18" alt="Pick another theme!"></button>
|
||
<div id="theme-choices"></div>
|
||
</div>
|
||
<!-- Theme Picker -->
|
||
|
||
<!-- End scripts/rustdoc-before.html -->
|
||
|
||
|
||
<h1 class="title">Automated Testing with Ox64 BL808 Emulator (Apache NuttX RTOS)</h1>
|
||
<nav id="rustdoc"><ul>
|
||
<li><a href="#scripting-the-expected" title="Scripting The Expected">1 Scripting The Expected</a><ul></ul></li>
|
||
<li><a href="#daily-automated-testing" title="Daily Automated Testing">2 Daily Automated Testing</a><ul></ul></li>
|
||
<li><a href="#boot-nuttx-in-supervisor-mode" title="Boot NuttX in Supervisor Mode">3 Boot NuttX in Supervisor Mode</a><ul></ul></li>
|
||
<li><a href="#emulate-the-system-timer" title="Emulate the System Timer">4 Emulate the System Timer</a><ul></ul></li>
|
||
<li><a href="#emulate-the-uart-interrupts" title="Emulate the UART Interrupts">5 Emulate the UART Interrupts</a><ul></ul></li>
|
||
<li><a href="#whats-next" title="What’s Next">6 What’s Next</a><ul></ul></li>
|
||
<li><a href="#appendix-boot-nuttx-in-supervisor-mode" title="Appendix: Boot NuttX in Supervisor Mode">7 Appendix: Boot NuttX in Supervisor Mode</a><ul></ul></li>
|
||
<li><a href="#appendix-start-the-system-timer" title="Appendix: Start the System Timer">8 Appendix: Start the System Timer</a><ul></ul></li>
|
||
<li><a href="#appendix-read-the-system-time" title="Appendix: Read the System Time">9 Appendix: Read the System Time</a><ul></ul></li>
|
||
<li><a href="#appendix-trigger-the-timer-interrupt" title="Appendix: Trigger the Timer Interrupt">10 Appendix: Trigger the Timer Interrupt</a><ul></ul></li></ul></nav><p>📝 <em>28 Jan 2024</em></p>
|
||
<p><img src="https://lupyuen.github.io/images/tinyemu3-title.jpg" alt="Automated Testing with Ox64 BL808 Emulator (Apache NuttX RTOS)" /></p>
|
||
<p><em>Every day we’re auto-building Apache NuttX RTOS for Ox64 BL808 SBC…</em></p>
|
||
<p><em>Can we test NuttX on Ox64 automatically after building?</em></p>
|
||
<p>Yes we can! With a little help from the <strong>Ox64 BL808 Emulator</strong> that we created last week…</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/tinyemu2"><strong>“Emulate Ox64 BL808 in the Web Browser: Experiments with TinyEMU RISC-V Emulator and Apache NuttX RTOS”</strong></a></li>
|
||
</ul>
|
||
<p><em>But our Ox64 Emulator was incomplete?</em></p>
|
||
<p>Today we fill in the missing pieces of our Ox64 Emulator and run it for <strong>Automated Testing</strong>…</p>
|
||
<ul>
|
||
<li>
|
||
<p>We boot NuttX in <strong>Supervisor Mode</strong></p>
|
||
<p>(Instead of Machine Mode)</p>
|
||
</li>
|
||
<li>
|
||
<p>Emulate OpenSBI for setting the <strong>System Timer</strong></p>
|
||
<p>(And read the System Time)</p>
|
||
</li>
|
||
<li>
|
||
<p>Emulate the UART Interrupts for <strong>Console Input</strong></p>
|
||
<p>(By modding the VirtIO Console)</p>
|
||
</li>
|
||
<li>
|
||
<p>Execute everything with <strong>Expect Scripting</strong></p>
|
||
<p>(Based on good old TCL)</p>
|
||
</li>
|
||
<li>
|
||
<p>Which becomes our <strong>Daily Automated Testing</strong></p>
|
||
<p>(Triggered every day by GitHub Actions)</p>
|
||
</li>
|
||
</ul>
|
||
<p>We begin with the easier bit: Scripting our Ox64 Emulator…</p>
|
||
<p><img src="https://lupyuen.github.io/images/tinyemu2-title.png" alt="Ox64 BL808 Emulator runs in a Web Browser too" /></p>
|
||
<p><a href="https://lupyuen.github.io/nuttx-tinyemu/timer"><em>Ox64 BL808 Emulator runs in a Web Browser too</em></a></p>
|
||
<h1 id="scripting-the-expected"><a class="doc-anchor" href="#scripting-the-expected">§</a>1 Scripting The Expected</h1>
|
||
<p><em>What’s this “Expect Scripting”?</em></p>
|
||
<p><a href="https://en.wikipedia.org/wiki/Expect"><strong><code>expect</code></strong></a> is a cool Command-Line Tool that sends commands to another app and checks the responses.</p>
|
||
<p><em>How is it used for Automated Testing?</em></p>
|
||
<p>Normally when we start our Ox64 Emulator, it boots NuttX and <strong>waits for our command</strong>…</p>
|
||
<div class="example-wrap"><pre class="language-text"><code>## Start our Ox64 Emulator with NuttX
|
||
$ ./temu nuttx.cfg
|
||
|
||
TinyEMU Emulator for Ox64 BL808 RISC-V SBC
|
||
NuttShell (NSH) NuttX-12.4.0-RC0
|
||
nsh></code></pre></div>
|
||
<p><a href="https://github.com/lupyuen/nuttx-ox64/blob/main/nuttx.cfg">(<strong>nuttx.cfg</strong> is our <strong>TinyEMU Config</strong>)</a></p>
|
||
<p>With an <strong>Expect Script</strong>, we can <strong>feed our commands automatically</strong> into the Emulator…</p>
|
||
<div class="example-wrap"><pre class="language-text"><code>## Run our Expect Script...
|
||
$ ./nuttx.exp
|
||
|
||
## Which starts the Ox64 Emulator...
|
||
spawn ./temu nuttx.cfg
|
||
|
||
## And sends a Command to the Emulator
|
||
nsh> uname -a
|
||
NuttX 12.4.0-RC0 55ec92e181 Jan 24 2024 00:11:08 risc-v ox64
|
||
nsh> </code></pre></div>
|
||
<p>That’s why we create an Expect Script to test Ox64 NuttX.</p>
|
||
<p><em>What’s nuttx.exp?</em></p>
|
||
<p>That’s our <strong>Expect Script</strong> containing the commands that will be sent to our Ox64 Emulator: <a href="https://github.com/lupyuen/nuttx-ox64/blob/main/nuttx.exp">nuttx.exp</a></p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>#!/usr/bin/expect
|
||
## Expect Script for Testing NuttX with Ox64 BL808 Emulator
|
||
|
||
## For every 1 character sent, wait 0.001 milliseconds
|
||
set send_slow {1 0.001}
|
||
|
||
## Start the Ox64 BL808 Emulator
|
||
spawn ./temu nuttx.cfg
|
||
|
||
## Wait for the prompt and enter `uname -a`
|
||
## `send -s` will send slowly (0.001 ms per char)
|
||
expect "nsh> "
|
||
send -s "uname -a\r"</code></pre></div>
|
||
<p><em>Will it work for complicated tests?</em></p>
|
||
<p>Yep we may use <strong>Pattern Matching</strong> and <strong>Timeout Detection</strong> in our script: <a href="https://github.com/lupyuen/nuttx-ox64/blob/main/nuttx.exp">nuttx.exp</a></p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>## Wait for the prompt and enter `ostest`
|
||
expect "nsh> "
|
||
send -s "ostest\r"
|
||
|
||
## Wait at most 30 seconds
|
||
set timeout 30
|
||
|
||
## Check the response...
|
||
expect {
|
||
## If we see this message, exit normally
|
||
"ostest_main: Exiting with status -1" { exit 0 }
|
||
|
||
## If timeout, exit with an error
|
||
timeout { exit 1 }
|
||
}</code></pre></div>
|
||
<p>Which works great for thoroughly exercising <strong>NuttX on our Ox64 Emulator</strong>…</p>
|
||
<div class="example-wrap"><pre class="language-text"><code>## Run our Expect Script to start Ox64 Emulator...
|
||
$ ./nuttx.exp
|
||
spawn ./temu nuttx.cfg
|
||
|
||
## And run all kinds of NuttX Tests
|
||
nsh> ostest
|
||
...
|
||
ostest_main: Exiting with status -1
|
||
nsh>
|
||
|
||
## Our Expect Script completes successfully</code></pre></div>
|
||
<p><a href="https://gist.github.com/lupyuen/1693ffb16ae943e44faada4428335eb0">(See the <strong>Test Log</strong>)</a></p>
|
||
<p><img src="https://lupyuen.github.io/images/tinyemu3-test.jpg" alt="NuttX for Ox64 gets tested every day" /></p>
|
||
<p><a href="https://github.com/lupyuen/nuttx-ox64/actions/workflows/ox64-test.yml"><em>NuttX for Ox64 gets tested every day</em></a></p>
|
||
<h1 id="daily-automated-testing"><a class="doc-anchor" href="#daily-automated-testing">§</a>2 Daily Automated Testing</h1>
|
||
<p><em>We run this every day?</em></p>
|
||
<p><strong>GitHub Actions</strong> will start our Automated Test every day at 12:55am (GMT): <a href="https://github.com/lupyuen/nuttx-ox64/blob/main/.github/workflows/ox64-test.yml">ox64-test.yml</a></p>
|
||
<div class="example-wrap"><pre class="language-yaml"><code>## Run our Automated Test
|
||
## Every day at 0:55 UTC
|
||
## (After Daily Build at 0:00 UTC)
|
||
on:
|
||
schedule:
|
||
- cron: '55 0 * * *'</code></pre></div>
|
||
<p><a href="https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule">(Why not one o’clock? <strong>It’s too busy</strong>)</a></p>
|
||
<p><em>What happens during Daily Automated Testing?</em></p>
|
||
<p>First it builds our <strong>Ox64 BL808 Emulator</strong>: <a href="https://github.com/lupyuen/nuttx-ox64/blob/main/.github/workflows/ox64-test.yml#L29-L58">ox64-test.yml</a></p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>## Install `expect` and the Build Prerequisites on Ubuntu
|
||
sudo apt -y update
|
||
sudo apt -y install \
|
||
expect libcurl4-openssl-dev libssl-dev zlib1g-dev libsdl2-dev wget
|
||
|
||
## Build our Ox64 BL808 Emulator
|
||
git clone https://github.com/lupyuen/ox64-tinyemu
|
||
pushd ox64-tinyemu
|
||
make
|
||
cp temu ..
|
||
popd</code></pre></div>
|
||
<p><strong>For macOS:</strong> We need extra steps…</p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>brew install openssl sdl2
|
||
make \
|
||
CFLAGS="-I$(brew --prefix)/opt/openssl/include -I$(brew --prefix)/opt/sdl2/include" \
|
||
LDFLAGS="-L$(brew --prefix)/opt/openssl/lib -L$(brew --prefix)/opt/sdl2/lib" \
|
||
CONFIG_MACOS=y</code></pre></div>
|
||
<p>Next we download the <a href="https://github.com/lupyuen/nuttx-ox64#nuttx-automated-daily-build-for-ox64"><strong>Daily NuttX Build</strong></a>…</p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>## Location of Daily NuttX Builds
|
||
## `outputs.date` looks like `2024-01-25`
|
||
url=https://github.com/lupyuen/nuttx-ox64/releases/download/nuttx-ox64-${{ steps.date.outputs.date }}
|
||
|
||
## Download the NuttX Build and print the Git Hash
|
||
wget $url/Image
|
||
wget $url/nuttx.hash
|
||
cat nuttx.hash
|
||
|
||
## Git Hash looks like...
|
||
## NuttX Source: https://github.com/apache/nuttx/tree/501896415589aa1a0264b0765746d8bdb43bdf42
|
||
## NuttX Apps: https://github.com/apache/nuttx-apps/tree/a16fb23dd752e84849ffcf865fc5d6d5ac745e43</code></pre></div>
|
||
<p><a href="https://github.com/lupyuen/nuttx-ox64/tags">(<strong>NuttX Builds</strong> are here)</a></p>
|
||
<p><a href="https://github.com/lupyuen/nuttx-ox64/blob/main/.github/workflows/ox64-test.yml#L25-L29">(<strong>outputs.date</strong> is defined here)</a></p>
|
||
<p>And we start our <strong>Test Script</strong>…</p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>## Download the Test Script from github.com/lupyuen/nuttx-ox64
|
||
url=https://github.com/lupyuen/nuttx-ox64/raw/main
|
||
wget $url/nuttx.cfg
|
||
wget $url/nuttx.exp
|
||
|
||
## Run the Test Script
|
||
chmod +x nuttx.exp
|
||
./nuttx.exp</code></pre></div>
|
||
<p><a href="https://github.com/lupyuen/nuttx-ox64/blob/main/nuttx.cfg">(<strong>nuttx.cfg</strong> is our <strong>TinyEMU Config</strong>)</a></p>
|
||
<p><a href="https://github.com/lupyuen/nuttx-ox64/blob/main/nuttx.exp">(<strong>nuttx.exp</strong> is our <strong>Expect Script</strong>)</a></p>
|
||
<p>That’s everything we need for Daily Automated Testing! Our Ox64 Emulator will emulate <a href="https://github.com/apache/nuttx-apps/blob/master/testing/ostest/ostest_main.c"><strong><code>ostest</code></strong></a> and launch a whole bunch of tests…</p>
|
||
<span style="font-size:90%">
|
||
<div><table><thead><tr><th style="text-align: left"></th><th style="text-align: left"></th><th style="text-align: left"></th></tr></thead><tbody>
|
||
<tr><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/blob/master/testing/ostest/ostest_main.c#L622-L639"><strong>Standard I/O</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/blob/master/testing/ostest/ostest_main.c#L146-L209"><strong>Environment Variables</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/setvbuf.c"><strong>Stream VBuf</strong></a></td></tr>
|
||
<tr><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/mutex.c"><strong>Mutex</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/cancel.c"><strong>Start Thread</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/robust.c"><strong>Robust Mutex</strong></a></td></tr>
|
||
<tr><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/sem.c"><strong>Semaphore</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/semtimed.c"><strong>Timed Semaphore</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/cond.c"><strong>Condition Variables</strong></a></td></tr>
|
||
<tr><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/pthread_exit.c"><strong>PThread Exit</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/timedwait.c"><strong>Timed Wait</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/mqueue.c"><strong>Message Queue</strong></a></td></tr>
|
||
<tr><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/timedmqueue.c"><strong>Timed Message Queue</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/sighand.c"><strong>Signal Handler</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/signest.c"><strong>Nested Signal Handler</strong></a></td></tr>
|
||
<tr><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/posixtimer.c"><strong>POSIX Timer</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/roundrobin.c"><strong>Round-Robin Scheduler</strong></a></td><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/barrier.c"><strong>PThread Barrier</strong></a></td></tr>
|
||
<tr><td style="text-align: left"><a href="https://github.com/apache/nuttx-apps/tree/master/testing/ostest/schedlock.c"><strong>Scheduler Lock</strong></a></td><td style="text-align: left"><a href="https://gist.github.com/lupyuen/1693ffb16ae943e44faada4428335eb0">(See the <strong>Test Log</strong>)</a></td><td style="text-align: left"><a href="https://github.com/lupyuen/nuttx-ox64/actions/workflows/ox64-test.yml">(See the <strong>Daily Logs</strong>)</a></td></tr>
|
||
</tbody></table>
|
||
</div></span>
|
||
<p><img src="https://lupyuen.github.io/images/tinyemu2-flow2.jpg" alt="NuttX Kernel won’t work in Machine Mode" /></p>
|
||
<h1 id="boot-nuttx-in-supervisor-mode"><a class="doc-anchor" href="#boot-nuttx-in-supervisor-mode">§</a>3 Boot NuttX in Supervisor Mode</h1>
|
||
<p><em>Automated Testing doesn’t look so hard?</em></p>
|
||
<p>That’s because we did the tough work inside our <a href="https://lupyuen.github.io/articles/tinyemu2"><strong>Ox64 BL808 Emulator</strong></a>! Let’s look back at the challenging bits…</p>
|
||
<p><em>What’s this Supervisor Mode? Why does it matter?</em></p>
|
||
<p>We created our Ox64 Emulator with the <a href="https://lupyuen.github.io/articles/tinyemu2"><strong>TinyEMU RISC-V Emulator</strong></a>. And TinyEMU boots NuttX in <strong>RISC-V Machine Mode</strong>. (Pic above)</p>
|
||
<p>Which won’t work because NuttX expects to run in <strong>RISC-V Supervisor Mode</strong>…</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/tinyemu2#machine-mode-vs-supervisor-mode"><strong>“Machine Mode vs Supervisor Mode”</strong></a></li>
|
||
</ul>
|
||
<p><em>All Operating Systems should boot in Machine Mode. Right?</em></p>
|
||
<p>Actually a <strong>RISC-V SBC</strong> (like Ox64) will boot the <a href="https://lupyuen.github.io/articles/sbi"><strong>OpenSBI Supervisor Binary Interface</strong></a> in <strong>Machine Mode</strong>…</p>
|
||
<p>Followed by the <a href="https://lupyuen.github.io/articles/ox2#appendix-nuttx-boot-flow"><strong>NuttX Kernel</strong></a> (or Linux Kernel) in <strong>Supervisor Mode</strong>…</p>
|
||
<p><img src="https://lupyuen.github.io/images/tinyemu2-flow.jpg" alt="Ox64 SBC will run in Machine, Supervisor AND User Modes" /></p>
|
||
<p><em>How to fix this?</em></p>
|
||
<p>We tweak TinyEMU to boot NuttX in <strong>Supervisor Mode</strong> (instead of Machine Mode)…</p>
|
||
<p><img src="https://lupyuen.github.io/images/tinyemu2-flow3.jpg" alt="TinyEMU will boot NuttX in Supervisor Mode" /></p>
|
||
<p>We do this in the <strong>TinyEMU Boot Code</strong>: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L874-L885">riscv_machine.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// At Startup: Init the TinyEMU Boot Code...
|
||
void copy_bios(...) {
|
||
...
|
||
// Load RAM_BASE_ADDR into Register T0.
|
||
// That's 0x5020_0000, the Start Address of
|
||
// NuttX Kernel (Linux too)
|
||
auipc t0, RAM_BASE_ADDR
|
||
|
||
// Load the Device Tree into Register A1.
|
||
// (Used by Linux but not NuttX)
|
||
auipc a1, dtb
|
||
addi a1, a1, dtb
|
||
|
||
// Load the Hart ID (CPU ID: 0) into Register A0
|
||
csrr a0, mhartid</code></pre></div>
|
||
<p>The code above comes from the original TinyEMU Emulator.</p>
|
||
<p>Next comes the code that we specially inserted for our <strong>Ox64 Emulator</strong>: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L882-L960">riscv_machine.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code> // Previously: We jump to RAM_BASE_ADDR in Machine Mode
|
||
// Now: We jump to RAM_BASE_ADDR in Supervisor Mode...
|
||
|
||
// Delegate all Exceptions to Supervisor Mode (instead of Machine Mode)
|
||
// We set MEDELEG CSR Register to 0xFFFF
|
||
lui a5, 0x10 ; nop // A5 is 0x10000
|
||
addiw a5, a5, -1 ; nop // A5 is 0xFFFF
|
||
csrw medeleg, a5
|
||
|
||
// Delegate all Interrupts to Supervisor Mode (instead of Machine Mode)
|
||
// We set MIDELEG CSR Register to 0xFFFF
|
||
csrw mideleg, a5
|
||
|
||
// Rightfully: Follow the OpenSBI Settings for Ox64
|
||
// Boot HART MIDELEG: 0x0222
|
||
// Boot HART MEDELEG: 0xB109</code></pre></div>
|
||
<p><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-boot-nuttx-in-supervisor-mode">(<strong>MEDELEG and MIDELEG</strong> explained)</a></p>
|
||
<p>The code above delegates all <strong>Exceptions and Interrupts</strong> to <strong>RISC-V Supervisor Mode</strong>. (Instead of Machine Mode)</p>
|
||
<p>Next we set the <strong>Previous Privilege Mode</strong> to Supervisor Mode (we’ll see why)…</p>
|
||
<div class="example-wrap"><pre class="language-c"><code> // Clear these bits in MSTATUS CSR Register...
|
||
// MPP (Bits 11 and 12): Clear the Previous Privilege Mode
|
||
lui a5, 0xffffe ; nop
|
||
addiw a5, a5, 2047
|
||
csrc mstatus, a5
|
||
|
||
// Set these bits in MSTATUS CSR Register...
|
||
// MPPS (Bit 11): Previous Privilege Mode is Supervisor Mode
|
||
// SUM (Bit 18): Allow Supervisor Mode to access Memory of User Mode
|
||
lui a5, 0x41
|
||
addiw a5, a5, -2048
|
||
csrs mstatus, a5</code></pre></div>
|
||
<p><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-boot-nuttx-in-supervisor-mode">(<strong>MSTATUS and SUM</strong> are explained here)</a></p>
|
||
<p>Why set Previous Privilege to Supervisor Mode? So we can execute an <strong>MRET (Return from Machine Mode)</strong> that will jump to the Previous Privilege… <strong>Supervisor Mode!</strong></p>
|
||
<div class="example-wrap"><pre class="language-c"><code> // Jump to RAM_BASE_ADDR in Supervisor Mode:
|
||
// Set the MEPC CSR Register to RAM_BASE_ADDR
|
||
// Then Return from Machine Mode to Supervisor Mode
|
||
csrw mepc, t0
|
||
mret</code></pre></div>
|
||
<p><em>Do we need so much Boot Code?</em></p>
|
||
<p>Yes! Check out what happens if we remove some bits of our Boot Code from TinyEMU…</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-boot-nuttx-in-supervisor-mode"><strong>“Boot NuttX in Supervisor Mode”</strong></a></li>
|
||
</ul>
|
||
<p><img src="https://lupyuen.github.io/images/tinyemu2-flow3.jpg" alt="TinyEMU will emulate the System Timer" /></p>
|
||
<h1 id="emulate-the-system-timer"><a class="doc-anchor" href="#emulate-the-system-timer">§</a>4 Emulate the System Timer</h1>
|
||
<p><em>NuttX can’t access the System Timer because it runs in RISC-V Supervisor Mode…</em></p>
|
||
<p><em>What can we do to help?</em></p>
|
||
<p>NuttX will make a <strong>System Call (ECALL)</strong> to OpenSBI to start the System Timer (pic above)…</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/nim#appendix-opensbi-timer-for-nuttx"><strong>“OpenSBI Timer for NuttX”</strong></a></li>
|
||
</ul>
|
||
<p>And NuttX reads the System Time through the <strong>TIME CSR Register</strong>: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/supervisor/riscv_sbi.c#L125-L127">riscv_sbi.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// Fetch the System Time...
|
||
uint64_t riscv_sbi_get_time(void) {
|
||
|
||
// Read the TIME CSR Register, which becomes
|
||
// the `RDTIME` RISC-V Instruction
|
||
return READ_CSR(time);
|
||
}</code></pre></div>
|
||
<p>Thus we emulate the <a href="https://lupyuen.github.io/articles/sbi#set-a-system-timer"><strong>OpenSBI System Timer</strong></a> and the <a href="https://five-embeddev.com/riscv-isa-manual/latest/counters.html#zicntr-standard-extension-for-base-counters-and-timers"><strong>TIME CSR Register</strong></a>.</p>
|
||
<p><strong>At Startup:</strong> We search for the ECALL to OpenSBI and remember the <strong>ECALL Address</strong>: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L916-L927">riscv_machine.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// Scan the Kernel Image for Special Instructions...
|
||
uint8_t *kernel_ptr = get_ram_ptr(s, RAM_BASE_ADDR, TRUE);
|
||
for (int i = 0; i < 0x10000; i++) {
|
||
|
||
// If we find the ECALL Instruction:
|
||
// 00000073 ecall
|
||
const uint8_t ecall[] = { 0x73, 0x00, 0x00, 0x00 };
|
||
if (memcmp(&kernel_ptr[i], ecall, sizeof(ecall)) == 0) {
|
||
|
||
// Remember the ECALL Address
|
||
ecall_addr = RAM_BASE_ADDR + i;
|
||
}</code></pre></div>
|
||
<p><em>What about the TIME CSR Register?</em></p>
|
||
<p>The <a href="https://five-embeddev.com/riscv-isa-manual/latest/counters.html#zicntr-standard-extension-for-base-counters-and-timers"><strong>TIME CSR Register</strong></a> gets assembled into the <a href="https://five-embeddev.com/riscv-isa-manual/latest/counters.html#zicntr-standard-extension-for-base-counters-and-timers"><strong>RDTIME RISC-V Instruction</strong></a>…</p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// Read the TIME CSR Register, which becomes
|
||
// the `RDTIME` RISC-V Instruction
|
||
nuttx/arch/risc-v/src/common/supervisor/riscv_sbi.c:126
|
||
riscv_sbi_get_time():
|
||
return READ_CSR(time);
|
||
5020bae6: c0102573 rdtime a0</code></pre></div>
|
||
<p><a href="https://github.com/lupyuen/nuttx-tinyemu/blob/main/docs/timer/nuttx.S">(See the <strong>NuttX Disassembly</strong>)</a></p>
|
||
<p>However <strong>RDTIME</strong> isn’t supported by TinyEMU. <a href="https://five-embeddev.com/riscv-isa-manual/latest/counters.html#zicntr-standard-extension-for-base-counters-and-timers">(Needs the <strong>Zicntr Extension</strong>)</a></p>
|
||
<p>Hence we patch <strong>RDTIME</strong> to become <strong>ECALL</strong>, and we emulate later: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L927-L937">riscv_machine.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code> // If we find the RDTIME Instruction: (Read System Time)
|
||
// c0102573 rdtime a0
|
||
const uint8_t rdtime[] = { 0x73, 0x25, 0x10, 0xc0 };
|
||
if (memcmp(&kernel_ptr[i], rdtime, sizeof(rdtime)) == 0) {
|
||
|
||
// Patch RDTIME to become ECALL:
|
||
// 00000073 ecall
|
||
const uint8_t ecall[] = { 0x73, 0x00, 0x00, 0x00 };
|
||
memcpy(&kernel_ptr[i], ecall, sizeof(ecall));
|
||
|
||
// Remember the RDTIME Address
|
||
rdtime_addr = RAM_BASE_ADDR + i;
|
||
}</code></pre></div>
|
||
<p>How to handle both ECALLs? Check the details here…</p>
|
||
<ul>
|
||
<li>
|
||
<p><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-start-the-system-timer"><strong>“Start the System Timer”</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-read-the-system-time"><strong>“Read the System Time”</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-trigger-the-timer-interrupt"><strong>“Trigger the Timer Interrupt”</strong></a></p>
|
||
</li>
|
||
</ul>
|
||
<p><em>Anything else we patched?</em></p>
|
||
<p>We patched these Special RISC-V Instructions to become ECALL: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L937-L956"><strong>DCACHE.IALL</strong> and <strong>SYNC.S</strong></a></p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>## Ox64 Emulator patches the Special Instructions
|
||
$ ./temu nuttx.cfg
|
||
|
||
TinyEMU Emulator for Ox64 BL808 RISC-V SBC
|
||
Patched DCACHE.IALL (Invalidate all Page Table Entries in the D-Cache)
|
||
at 0x5020099a
|
||
Patched SYNC.S (Ensure that all Cache Operations are completed)
|
||
at 0x5020099e
|
||
Found ECALL (Start System Timer)
|
||
at 0x5020bae0
|
||
Patched RDTIME (Read System Time)
|
||
at 0x5020bae6</code></pre></div>
|
||
<p><a href="https://gist.github.com/lupyuen/1693ffb16ae943e44faada4428335eb0">(See the <strong>Test Log</strong>)</a></p>
|
||
<p><a href="https://github.com/lupyuen/nuttx-tinyemu/blob/main/docs/timer/nuttx.S">(See the <strong>NuttX Disassembly</strong>)</a></p>
|
||
<p>These instructions are specific to <strong>T-Head C906 CPU</strong> (and won’t work in TinyEMU). NuttX calls them to <a href="https://lupyuen.github.io/articles/mmu#appendix-flush-the-mmu-cache-for-t-head-c906"><strong>Flush the MMU Cache</strong></a>.</p>
|
||
<p>(Though we don’t emulate them right now)</p>
|
||
<p><img src="https://lupyuen.github.io/images/plic2-registers.jpg" alt="UART Interrupts for Ox64 BL808 SBC" /></p>
|
||
<h1 id="emulate-the-uart-interrupts"><a class="doc-anchor" href="#emulate-the-uart-interrupts">§</a>5 Emulate the UART Interrupts</h1>
|
||
<p><em>Ox64 SBC has a UART Controller that will handle Console Input…</em></p>
|
||
<p><em>How do we emulate the Ox64 UART Controller?</em></p>
|
||
<p>Previously we emulated the <strong>BL808 UART Registers</strong> to do Console Output…</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/tinyemu2#intercept-the-uart-registers"><strong>“Intercept the UART Registers”</strong></a></li>
|
||
</ul>
|
||
<p>Console Input is tricky… We need to emulate <strong>UART Interrupts</strong>! (Pic above)</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/plic2"><strong>“UART Interrupt and Platform-Level Interrupt Controller”</strong></a></li>
|
||
</ul>
|
||
<p><em>Any UART Controller in TinyEMU that we can reuse?</em></p>
|
||
<p>TinyEMU has a <a href="https://lupyuen.github.io/articles/tinyemu#virtio-console"><strong>VirtIO Console</strong></a> that works like a UART Controller.</p>
|
||
<p>Let’s hack TinyEMU’s VirtIO Console so that it behaves like <a href="https://lupyuen.github.io/articles/plic2#appendix-uart-driver-for-ox64"><strong>BL808 UART Controller</strong></a>.</p>
|
||
<p>We tweak the <strong>VirtIO Interrupt Number</strong> so it triggers the same Interrupt as BL808 UART3: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L69-L85">riscv_machine.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// VirtIO now emulates
|
||
// BL808 UART3 Interrupt
|
||
#define VIRTIO_IRQ 20</code></pre></div>
|
||
<p>When we detect a keypress, we trigger the <strong>UART Interrupt</strong>: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/virtio.c#L1338-L1347">virtio.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// When we receive a keypress...
|
||
int virtio_console_write_data(VIRTIODevice *s, const uint8_t *buf, int buf_len) {
|
||
|
||
// Pass the keypress to NuttX later
|
||
set_input(buf[0]);
|
||
|
||
// Trigger the UART Interrupt
|
||
s->int_status |= 1;
|
||
set_irq(s->irq, 1);</code></pre></div>
|
||
<p><a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/virtio.c#L2697-L2704">(<strong>set_input</strong> is defined here)</a></p>
|
||
<p><a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L319-L332">(<strong>set_irq</strong> is defined here)</a></p>
|
||
<p>When we run this: TinyEMU loops forever handling UART Interrupts :-(</p>
|
||
<p><em>Surely we need to Clear the UART Interrupt?</em></p>
|
||
<p>We check our <strong>BL808 UART Driver</strong> in NuttX: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_serial.c#L166-L224">bl808_serial.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// NuttX Interrupt Handler for BL808 UART
|
||
int uart_interrupt(int irq, void *context, void *arg) {
|
||
|
||
// At 0x3000_2020: Read the UART Interrupt Status (uart_int_sts)
|
||
int_status = getreg32(BL808_UART_INT_STS(uart_idx));
|
||
|
||
// At 0x3000_2024: Read the UART Interrupt Mask (uart_int_mask)
|
||
int_mask = getreg32(BL808_UART_INT_MASK(uart_idx));
|
||
|
||
// If UART Interrupt Status says there's UART Input...
|
||
if ((int_status & UART_INT_STS_URX_END_INT) &&
|
||
!(int_mask & UART_INT_MASK_CR_URX_END_MASK)) {
|
||
|
||
// At 0x3000_2028: Clear the UART Interrupt (uart_int_clear)
|
||
putreg32(UART_INT_CLEAR_CR_URX_END_CLR, BL808_UART_INT_CLEAR(uart_idx));
|
||
|
||
// At 0x3000_208C: Read the UART Input (uart_fifo_rdata)
|
||
uart_recvchars(dev);</code></pre></div>
|
||
<p><a href="https://github.com/apache/nuttx/blob/master/drivers/serial/serial_io.c#L107-L268">(<strong>uart_recvchars</strong> is defined here)</a></p>
|
||
<p>Aha! We must emulate the <strong>BL808 UART Registers</strong> above…</p>
|
||
<ol>
|
||
<li>
|
||
<p><a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_cpu.c#L402-L407"><strong>UART Interrupt Status</strong></a> should say there’s UART Input</p>
|
||
<p><a href="https://github.com/bouffalolab/bl_docs/blob/main/BL808_RM/en/BL808_RM_en_1.3.pdf">(<strong>uart_int_sts</strong>, Page 419)</a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_cpu.c#L407-L412"><strong>UART Interrupt Mask</strong></a> should return 0</p>
|
||
<p><a href="https://github.com/bouffalolab/bl_docs/blob/main/BL808_RM/en/BL808_RM_en_1.3.pdf">(<strong>uart_int_mask</strong>, Page 420)</a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_cpu.c#L526-L532"><strong>UART Clear Interrupt</strong></a> should clear the VirtIO Interrupt</p>
|
||
<p><a href="https://github.com/bouffalolab/bl_docs/blob/main/BL808_RM/en/BL808_RM_en_1.3.pdf">(<strong>uart_int_clear</strong>, Page 421)</a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_cpu.c#L412-L422"><strong>UART Input</strong></a> should return the keypress</p>
|
||
<p><a href="https://github.com/bouffalolab/bl_docs/blob/main/BL808_RM/en/BL808_RM_en_1.3.pdf">(<strong>uart_fifo_rdata</strong>, Page 428)</a></p>
|
||
</li>
|
||
</ol>
|
||
<p>Now we see NuttX correctly handling the UART Interrupt triggered by TinyEMU…</p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>## When we press a key...
|
||
## TinyEMU triggers the UART Interrupt
|
||
plic_set_irq: irq_num=20, state=1
|
||
plic_update_mip: set_mip, pending=0x80000, served=0x0
|
||
raise_exception: cause=-2147483639
|
||
raise_exception2: cause=-2147483639, tval=0x0
|
||
|
||
## NuttX Claims the UART Interrupt
|
||
plic_read: offset=0x201004
|
||
plic_update_mip: reset_mip, pending=0x80000, served=0x80000
|
||
|
||
## NuttX handles the UART Interrupt in Interrupt Handler
|
||
virtio_ack_irq
|
||
plic_set_irq: irq_num=20, state=0
|
||
plic_update_mip: reset_mip, pending=0x0, served=0x80000
|
||
|
||
## NuttX Completes the UART Interrupt
|
||
plic_write: offset=0x201004, val=0x14
|
||
plic_update_mip: reset_mip, pending=0x0, served=0x0</code></pre></div>
|
||
<p><a href="https://gist.github.com/lupyuen/de071bf54b603f4aaff3954648dcc340#file-ox64-tinyemu-log-L129-L172">(See the <strong>Complete Log</strong>)</a></p>
|
||
<p>Finally Console Input works OK yay!</p>
|
||
<ul>
|
||
<li>
|
||
<p><a href="https://lupyuen.github.io/nuttx-tinyemu/timer">Live Demo of <strong>Ox64 BL808 Emulator</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://youtu.be/FAxaMt6A59I">Watch the <strong>Demo on YouTube</strong></a></p>
|
||
</li>
|
||
</ul>
|
||
<p>Some more tweaks to TinyEMU VirtIO for Console Input…</p>
|
||
<ol>
|
||
<li>
|
||
<p>We always allow <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/virtio.c#L1297-L1313"><strong>VirtIO to Write Data</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p>We’re always <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/virtio.c#L1313-L1338"><strong>Ready for VirtIO Writes</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p>We disable the <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/virtio.c#L1370-L1382"><strong>Console Resize Event</strong></a></p>
|
||
<p>(Because the UART Interrupt crashes NuttX at startup)</p>
|
||
</li>
|
||
</ol>
|
||
<h1 id="whats-next"><a class="doc-anchor" href="#whats-next">§</a>6 What’s Next</h1>
|
||
<p>We created a tool that’s super helpful for <strong>validating our Daily NuttX Builds</strong>, checking if they’ll actually boot OK on Ox64…</p>
|
||
<ul>
|
||
<li>
|
||
<p>We created an <strong>Ox64 Emulator</strong> with TinyEMU RISC-V Emulator</p>
|
||
</li>
|
||
<li>
|
||
<p>Fixed it to boot NuttX in <strong>Supervisor Mode</strong></p>
|
||
</li>
|
||
<li>
|
||
<p>Emulated OpenSBI for setting the <strong>System Timer</strong></p>
|
||
</li>
|
||
<li>
|
||
<p>Emulated the UART Interrupts for <strong>Console Input</strong></p>
|
||
</li>
|
||
<li>
|
||
<p>Then we ran everything with an <strong>Expect Script</strong></p>
|
||
</li>
|
||
<li>
|
||
<p>Which becomes our <strong>Daily Automated Testing</strong></p>
|
||
</li>
|
||
</ul>
|
||
<p>Previously we tried creating a <a href="https://lupyuen.github.io/articles/unicorn2"><strong>PinePhone Emulator</strong></a>, but Arm64 Emulation was way too difficult. Ox64 with RISC-V is so much easier!</p>
|
||
<p>Many Thanks to my <a href="https://lupyuen.github.io/articles/sponsor"><strong>GitHub Sponsors</strong></a> (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.</p>
|
||
<ul>
|
||
<li>
|
||
<p><a href="https://lupyuen.github.io/articles/sponsor"><strong>Sponsor me a coffee</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://news.ycombinator.com/item?id=39160763"><strong>Discuss this article on Hacker News</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://bbs.bouffalolab.com/d/277-article-automated-testing-with-ox64-bl808-emulator-apache-nuttx-rtos"><strong>Discuss this article on Bouffalo Lab Forum</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://github.com/lupyuen/nuttx-ox64"><strong>My Current Project: “Apache NuttX RTOS for Ox64 BL808”</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://github.com/lupyuen/nuttx-star64"><strong>My Other Project: “NuttX for Star64 JH7110”</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://github.com/lupyuen/pinephone-nuttx"><strong>Older Project: “NuttX for PinePhone”</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://lupyuen.github.io"><strong>Check out my articles</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://lupyuen.github.io/rss.xml"><strong>RSS Feed</strong></a></p>
|
||
</li>
|
||
</ul>
|
||
<p><em>Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…</em></p>
|
||
<p><a href="https://github.com/lupyuen/lupyuen.github.io/blob/master/src/tinyemu3.md"><strong>lupyuen.github.io/src/tinyemu3.md</strong></a></p>
|
||
<p><img src="https://lupyuen.github.io/images/tinyemu2-flow3.jpg" alt="TinyEMU will boot NuttX in Supervisor Mode" /></p>
|
||
<h1 id="appendix-boot-nuttx-in-supervisor-mode"><a class="doc-anchor" href="#appendix-boot-nuttx-in-supervisor-mode">§</a>7 Appendix: Boot NuttX in Supervisor Mode</h1>
|
||
<p>Earlier we saw a big chunk of <strong>TinyEMU Boot Code</strong> (pic above) that will start <strong>NuttX in RISC-V Supervisor Mode</strong> (instead of Machine Mode)…</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/tinyemu3#boot-nuttx-in-supervisor-mode"><strong>“Boot NuttX in Supervisor Mode”</strong></a></li>
|
||
</ul>
|
||
<p><em>Can’t we call MRET directly? And jump from Machine Mode to Supervisor Mode?</em></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// Load RAM_BASE_ADDR into Register T0.
|
||
// That's 0x5020_0000, the Start Address of
|
||
// NuttX Kernel (Linux too)
|
||
auipc t0, RAM_BASE_ADDR
|
||
|
||
// Testing: Can we jump like this?
|
||
// Jump to RAM_BASE_ADDR in Supervisor Mode:
|
||
// Set the MEPC CSR Register to RAM_BASE_ADDR
|
||
// Then Return from Machine Mode to Supervisor Mode
|
||
csrw mepc, t0
|
||
mret</code></pre></div>
|
||
<p>Watch what happens when we run it…</p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>## Illegal Instruction in RISC-V User Mode (priv=U)
|
||
raise_exception2: cause=2
|
||
tval=0x10401073
|
||
pc=50200074
|
||
priv=U
|
||
mstatus=a00000080</code></pre></div>
|
||
<p>TinyEMU halts with an <strong>Illegal Instuction</strong>. The offending code is here: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_head.S#L121-L128">bl808_head.S</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>nuttx/arch/risc-v/src/chip/bl808_head.S:124
|
||
/* Disable all interrupts (i.e. timer, external) in sie */
|
||
csrw sie, zero
|
||
50200074: 10401073 csrw sie, zero</code></pre></div>
|
||
<p>“<strong><code>csrw sie</code></strong>” writes to SIE (Supervisor-Mode Interrupt Enable). And SIE is a <strong>Supervisor-Mode</strong> CSR Register.</p>
|
||
<p><em>Why is this instruction invalid?</em></p>
|
||
<p>The instruction is invalid because we’re running in <strong>RISC-V User Mode</strong> (<strong><code>priv=U</code></strong>), not Supervisor Mode!</p>
|
||
<p>Somehow <strong>MRET</strong> has jumped from Machine Mode to <strong>User Mode</strong>.</p>
|
||
<p>To fix this, we set the <strong>Previous Privilege Mode</strong> to Supervisor Mode (so MRET will jump to Supervisor Mode)…</p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// Clear these bits in MSTATUS CSR Register...
|
||
// MPP (Bits 11 and 12): Clear the Previous Privilege Mode
|
||
lui a5, 0xffffe ; nop
|
||
addiw a5, a5, 2047
|
||
csrc mstatus, a5
|
||
|
||
// Set these bits in MSTATUS CSR Register...
|
||
// MPPS (Bit 11): Previous Privilege Mode is Supervisor Mode
|
||
// SUM (Bit 18): Allow Supervisor Mode to access Memory of User Mode
|
||
lui a5, 0x41
|
||
addiw a5, a5, -2048
|
||
csrs mstatus, a5
|
||
|
||
// Return from Machine Mode to Supervisor Mode
|
||
mret</code></pre></div>
|
||
<p><a href="https://five-embeddev.com/riscv-isa-manual/latest/machine.html#machine-status-registers-mstatus-and-mstatush">(<strong>MSTATUS</strong> is explained here)</a></p>
|
||
<p><a href="https://lupyuen.github.io/articles/app#kernel-accesses-app-memory">(<strong>SUM</strong> is needed for NuttX Apps)</a></p>
|
||
<p><a href="https://gist.github.com/lupyuen/368744ef01b7feba10c022cd4f4c5ef2#file-nuttx-start-s-L1282-L1314">(Why <strong>Register A5</strong>? Because we copied from the <strong>NuttX QEMU Boot Code</strong>)</a></p>
|
||
<p>(Why <strong>NOP</strong>? Because TinyEMU needs every Boot Instruction padded to 32 bits)</p>
|
||
<p><em>Now what happens?</em></p>
|
||
<p>NuttX Shell makes a <strong>System Call (ECALL)</strong> to NuttX Kernel. Which is supposed to jump from RISC-V User Mode to <strong>Supervisor Mode</strong>…</p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>## NuttX Kernel starts NuttX Shell
|
||
nx_start_application: Starting init task: /system/bin/init
|
||
|
||
## NuttX Shell makes an ECALL from User Mode (priv=U)
|
||
raise_exception2: cause=8, tval=0x0
|
||
pc=800019c6
|
||
priv=U
|
||
mstatus=a000400a1
|
||
mideleg=00
|
||
mie=00
|
||
mip=80
|
||
|
||
## But TinyEMU jumps to Machine Mode! (priv=M)
|
||
raise_exception2: cause=2, tval=0x0
|
||
pc=00
|
||
priv=M
|
||
mstatus=a000400a1
|
||
mideleg=00
|
||
mie=00
|
||
mip=80</code></pre></div>
|
||
<p>Nope, it actually jumps from RISC-V User Mode (<strong><code>priv=U</code></strong>) to <strong>Machine Mode</strong> (<strong><code>priv=M</code></strong>)! (Instead of Supervisor Mode)</p>
|
||
<p>To fix this: We delegate all <strong>Exceptions and Interrupts</strong> to <strong>RISC-V Supervisor Mode</strong>. (Instead of Machine Mode)</p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// Delegate all Exceptions to Supervisor Mode (instead of Machine Mode)
|
||
// We set MEDELEG CSR Register to 0xFFFF
|
||
lui a5, 0x10 ; nop // A5 is 0x10000
|
||
addiw a5, a5, -1 ; nop // A5 is 0xFFFF
|
||
csrw medeleg, a5
|
||
|
||
// Delegate all Interrupts to Supervisor Mode (instead of Machine Mode)
|
||
// We set MIDELEG CSR Register to 0xFFFF
|
||
csrw mideleg, a5
|
||
|
||
// Rightfully: Follow the OpenSBI Settings for Ox64
|
||
// Boot HART MIDELEG: 0x0222
|
||
// Boot HART MEDELEG: 0xB109</code></pre></div>
|
||
<p>(<strong>MEDELEG</strong> is the Machine Exception Delegation Register)</p>
|
||
<p>(<strong>MIDELEG</strong> is the Machine Interrupt Delegation Register)</p>
|
||
<p><a href="https://five-embeddev.com/riscv-isa-manual/latest/machine.html#machine-trap-delegation-registers-medeleg-and-mideleg">(<strong>MEDELEG and MIDELEG</strong> are explained here)</a></p>
|
||
<p><em>Does it work?</em></p>
|
||
<p>Finally NuttX Shell starts OK! And User-Mode ECALLs are working perfectly yay!</p>
|
||
<div class="example-wrap"><pre class="language-text"><code>nx_start_application:
|
||
Starting init task:
|
||
/system/bin/init
|
||
NuttShell (NSH) NuttX-12.4.0
|
||
nsh></code></pre></div>
|
||
<p><a href="https://gist.github.com/lupyuen/de071bf54b603f4aaff3954648dcc340">(See the <strong>Complete Log</strong>)</a></p>
|
||
<p>That’s why we need the big chunk of <a href="https://lupyuen.github.io/articles/tinyemu3#boot-nuttx-in-supervisor-mode"><strong>TinyEMU Boot Code</strong></a> that we saw earlier.</p>
|
||
<p><img src="https://lupyuen.github.io/images/tinyemu2-flow3.jpg" alt="TinyEMU will emulate the System Timer" /></p>
|
||
<h1 id="appendix-start-the-system-timer"><a class="doc-anchor" href="#appendix-start-the-system-timer">§</a>8 Appendix: Start the System Timer</h1>
|
||
<p>Earlier we talked about emulating OpenSBI to <strong>start the System Timer</strong> (pic above)…</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/tinyemu3#emulate-the-system-timer"><strong>“Emulate the System Timer”</strong></a></li>
|
||
</ul>
|
||
<p>And at startup, we captured the address of the <strong>System Call (ECALL)</strong> from NuttX Kernel (Supervisor Mode) to OpenSBI (Machine Mode): <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L916-L927">riscv_machine.c</a></p>
|
||
<div class="example-wrap"><pre class="language-bash"><code>## Ox64 Emulator patches the Special Instructions
|
||
$ ./temu nuttx.cfg
|
||
|
||
TinyEMU Emulator for Ox64 BL808 RISC-V SBC
|
||
Patched DCACHE.IALL (Invalidate all Page Table Entries in the D-Cache)
|
||
at 0x5020099a
|
||
Patched SYNC.S (Ensure that all Cache Operations are completed)
|
||
at 0x5020099e
|
||
Found ECALL (Start System Timer)
|
||
at 0x5020bae0
|
||
Patched RDTIME (Read System Time)
|
||
at 0x5020bae6</code></pre></div>
|
||
<p><a href="https://gist.github.com/lupyuen/1693ffb16ae943e44faada4428335eb0">(See the <strong>Test Log</strong>)</a></p>
|
||
<p><a href="https://github.com/lupyuen/nuttx-tinyemu/blob/main/docs/timer/nuttx.S">(See the <strong>NuttX Disassembly</strong>)</a></p>
|
||
<p>This is how we emulate the <strong>ECALL to OpenSBI</strong>: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_cpu.c#L1164-L1182">riscv_cpu.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// Called by TinyEMU to handle RISC-V Exceptions
|
||
void raise_exception2(RISCVCPUState *s, uint32_t cause, target_ulong tval) {
|
||
...
|
||
// If this is an ECALL from Supervisor Mode...
|
||
// (Not ECALL from User Mode)
|
||
if (cause == CAUSE_SUPERVISOR_ECALL) {
|
||
|
||
// If Program Counter is the
|
||
// ECALL to OpenSBI...
|
||
if (s->pc == ecall_addr) {
|
||
|
||
// We emulate the OpenSBI Set Timer Function:
|
||
// https://github.com/riscv-non-isa/riscv-sbi-doc/blob/v1.0.0/riscv-sbi.adoc#61-function-set-timer-fid-0
|
||
|
||
// Clear the Pending Timer Interrupt Bit
|
||
// (Says the SBI Spec)
|
||
riscv_cpu_reset_mip(s, MIP_STIP);
|
||
|
||
// If Parameter A0 is not -1, set the System Timer (timecmp)
|
||
// (Parameter A0 is Register X10)
|
||
uint64_t timecmp = s->reg[10];
|
||
if (timecmp != (uint64_t) -1) {
|
||
set_timecmp(NULL, timecmp);
|
||
}
|
||
|
||
// Skip to the next instruction (RET)
|
||
s->pc += 4;
|
||
return; </code></pre></div>
|
||
<p><strong>set_timecmp</strong> sets the <strong>Machine-Mode System Timer</strong>: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L1225-L1235">riscv_machine.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// Set the System Timer
|
||
void set_timecmp(RISCVMachine *machine0, uint64_t timecmp) {
|
||
|
||
// At Startup: Remember the RISC-V Machine and return
|
||
static RISCVMachine *machine = NULL;
|
||
if (machine0 != NULL) { machine = machine0; return; }
|
||
|
||
// Otherwise set the System Timer
|
||
if (machine == NULL) { puts("set_timecmp: machine is null"); return; }
|
||
machine->timecmp = timecmp;
|
||
}</code></pre></div>
|
||
<p><a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L1136-L1140">(<strong>set_timecmp</strong> is initialised by <strong>riscv_machine_init</strong>)</a></p>
|
||
<p>Note that nothing will happen unless we trigger a <strong>Supervisor-Mode Timer Interrupt</strong> to NuttX…</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-trigger-the-timer-interrupt"><strong>“Trigger the Timer Interrupt”</strong></a></li>
|
||
</ul>
|
||
<p><em>We’re emulating the OpenSBI System Timer with the Machine-Mode System Timer?</em></p>
|
||
<p>Exactly! We do the same for reading the System Time…</p>
|
||
<h1 id="appendix-read-the-system-time"><a class="doc-anchor" href="#appendix-read-the-system-time">§</a>9 Appendix: Read the System Time</h1>
|
||
<p>Just now we talked about emulating the RDTIME RISC-V Instruction for <strong>reading the System Time</strong>…</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/tinyemu3#emulate-the-system-timer"><strong>“Emulate the System Timer”</strong></a></li>
|
||
</ul>
|
||
<p>And at startup we do these: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L927-L937">riscv_machine.c</a></p>
|
||
<ul>
|
||
<li>
|
||
<p>Capture the address of the <strong>RDTIME Instruction</strong></p>
|
||
</li>
|
||
<li>
|
||
<p>Patch the RDTIME Instruction to become a <strong>System Call (ECALL)</strong></p>
|
||
</li>
|
||
</ul>
|
||
<div class="example-wrap"><pre class="language-bash"><code>## Ox64 Emulator patches the Special Instructions
|
||
$ ./temu nuttx.cfg
|
||
|
||
TinyEMU Emulator for Ox64 BL808 RISC-V SBC
|
||
Patched DCACHE.IALL (Invalidate all Page Table Entries in the D-Cache)
|
||
at 0x5020099a
|
||
Patched SYNC.S (Ensure that all Cache Operations are completed)
|
||
at 0x5020099e
|
||
Found ECALL (Start System Timer)
|
||
at 0x5020bae0
|
||
Patched RDTIME (Read System Time)
|
||
at 0x5020bae6</code></pre></div>
|
||
<p><a href="https://gist.github.com/lupyuen/1693ffb16ae943e44faada4428335eb0">(See the <strong>Test Log</strong>)</a></p>
|
||
<p><a href="https://github.com/lupyuen/nuttx-tinyemu/blob/main/docs/timer/nuttx.S">(See the <strong>NuttX Disassembly</strong>)</a></p>
|
||
<p>This is how we emulate the Patched ECALL to <strong>read the System Time</strong>: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_cpu.c#L1183-L1195">riscv_cpu.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// Called by TinyEMU to handle RISC-V Exceptions
|
||
void raise_exception2(RISCVCPUState *s, uint32_t cause, target_ulong tval) {
|
||
...
|
||
// If this is an ECALL from Supervisor Mode...
|
||
// (Not ECALL from User Mode)
|
||
if (cause == CAUSE_SUPERVISOR_ECALL) {
|
||
|
||
// If Program Counter is the
|
||
// (formerly) RDTIME Instruction...
|
||
if (s->pc == rdtime_addr) {
|
||
|
||
// We emulate the RDTIME Instruction to fetch the System Time:
|
||
// https://five-embeddev.com/riscv-isa-manual/latest/counters.html#zicntr-standard-extension-for-base-counters-and-timers
|
||
|
||
// Return the System Time in Register A0
|
||
// (Which is aliased to Register X10)
|
||
s->reg[10] = real_time;
|
||
|
||
// Skip to the next instruction (RET)
|
||
s->pc += 4;
|
||
return; </code></pre></div>
|
||
<p><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-start-the-system-timer">(<strong>set_timecmp</strong> is explained here)</a></p>
|
||
<p><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-trigger-the-timer-interrupt">(<strong>real_time</strong> is explained in the next section)</a></p>
|
||
<p>Note that nothing will happen unless we trigger a <strong>Supervisor-Mode Timer Interrupt</strong> to NuttX…</p>
|
||
<ul>
|
||
<li><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-trigger-the-timer-interrupt"><strong>“Trigger the Timer Interrupt”</strong></a></li>
|
||
</ul>
|
||
<h1 id="appendix-trigger-the-timer-interrupt"><a class="doc-anchor" href="#appendix-trigger-the-timer-interrupt">§</a>10 Appendix: Trigger the Timer Interrupt</h1>
|
||
<p>Previously we discussed the emulation of the <strong>System Timer</strong>…</p>
|
||
<ul>
|
||
<li>
|
||
<p><a href="https://lupyuen.github.io/articles/tinyemu3#emulate-the-system-timer"><strong>“Emulate the System Timer”</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-start-the-system-timer"><strong>“Start the System Timer”</strong></a></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="https://lupyuen.github.io/articles/tinyemu3#appendix-read-the-system-time"><strong>“Read the System Time”</strong></a></p>
|
||
</li>
|
||
</ul>
|
||
<p>But nothing will happen unless we trigger a <strong>Supervisor-Mode Timer Interrupt</strong> to NuttX!</p>
|
||
<p>This is how we trigger the <strong>Timer Interrupt</strong>: <a href="https://github.com/lupyuen/ox64-tinyemu/blob/main/riscv_machine.c#L1172-L1182">riscv_machine.c</a></p>
|
||
<div class="example-wrap"><pre class="language-c"><code>// Called by TinyEMU periodically to check the System Timer
|
||
static int riscv_machine_get_sleep_duration(VirtMachine *s1, int delay) {
|
||
...
|
||
// Pass the System Time to raise_exception2()
|
||
real_time = rtc_get_time(m);
|
||
|
||
// If the Timer Interrupt has not been triggered...
|
||
if (!(riscv_cpu_get_mip(s) & MIP_STIP)) {
|
||
|
||
// And the System Timer has expired...
|
||
const int64_t delay2 = m->timecmp - rtc_get_time(m);
|
||
if (delay2 <= 0) {
|
||
|
||
// We trigger the Timer Interrupt
|
||
// for Supervisor Mode
|
||
riscv_cpu_set_mip(s, MIP_STIP);
|
||
}
|
||
}</code></pre></div>
|
||
<p>Again we’re reusing the <strong>Machine-Mode System Timer</strong>, to trigger the Supervisor-Mode Timer Interrupt.</p>
|
||
<p>With this Timer Interrupt, <strong><code>usleep</code></strong> (and other Timer Functions) will work perfectly in NuttX…</p>
|
||
<div class="example-wrap"><pre class="language-text"><code>Loading...
|
||
TinyEMU Emulator for Ox64 BL808 RISC-V SBC
|
||
NuttShell (NSH) NuttX-12.4.0-RC0
|
||
nsh> usleep 1
|
||
nsh> </code></pre></div>
|
||
<p><a href="https://gist.github.com/lupyuen/31bde9c2563e8ea2f1764fb95c6ea0fc">(See the <strong>Timer Log</strong>)</a></p>
|
||
|
||
|
||
<!-- Begin scripts/rustdoc-after.html: Post-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
|
||
|
||
<!-- Begin Theme Picker and Prism Theme -->
|
||
<script src="../theme.js"></script>
|
||
<script src="../prism.js"></script>
|
||
<!-- Theme Picker and Prism Theme -->
|
||
|
||
<!-- End scripts/rustdoc-after.html -->
|
||
|
||
|
||
</body>
|
||
</html> |