lupyuen/lupyuen.org
1
0
Fork 0
mirror of https://github.com/lupyuen/lupyuen.github.io.git synced 2025-01-13 09:08:30 +08:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
lupyuen.org/articles/app.html
Lup Yuen Lee 6e2f142414
Some checks are pending
Build Articles / build (push) Waiting to run
Details
Commit from GitHub Actions
2025-01-04 06:09:02 +00:00

964 lines
No EOL
55 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="rustdoc">
<title>RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk</title>
<!-- Begin scripts/articles/*-header.html: Article Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<meta property="og:title"
content="RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk"
data-rh="true">
<meta property="og:description"
content="(1) What's inside an Application for Apache NuttX RTOS (2) How it calls the NuttX Kernel (3) How NuttX Apps are bundled into the Initial RAM Disk for Pine64 Ox64 BL808 RISC-V SBC"
data-rh="true">
<meta name="description"
content="(1) What's inside an Application for Apache NuttX RTOS (2) How it calls the NuttX Kernel (3) How NuttX Apps are bundled into the Initial RAM Disk for Pine64 Ox64 BL808 RISC-V SBC">
<meta property="og:image"
content="https://lupyuen.github.io/images/app-title.png">
<meta property="og:type"
content="article" data-rh="true">
<link rel="canonical"
href="https://lupyuen.org/articles/app.html" />
<!-- End scripts/articles/*-header.html -->
<!-- Begin scripts/rustdoc-header.html: Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<link rel="alternate" type="application/rss+xml" title="RSS Feed for lupyuen" href="/rss.xml" />
<link rel="stylesheet" type="text/css" href="../normalize.css">
<link rel="stylesheet" type="text/css" href="../rustdoc.css" id="mainThemeStyle">
<link rel="stylesheet" type="text/css" href="../dark.css">
<link rel="stylesheet" type="text/css" href="../light.css" id="themeStyle">
<link rel="stylesheet" type="text/css" href="../prism.css">
<script src="../storage.js"></script><noscript>
<link rel="stylesheet" href="../noscript.css"></noscript>
<link rel="shortcut icon" href="../favicon.ico">
<style type="text/css">
#crate-search {
background-image: url("../down-arrow.svg");
}
</style>
<!-- End scripts/rustdoc-header.html -->
</head>
<body class="rustdoc">
<!--[if lte IE 8]>
<div class="warning">
This old browser is unsupported and will most likely display funky
things.
</div>
<![endif]-->
<!-- Begin scripts/rustdoc-before.html: Pre-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker -->
<div class="theme-picker" style="left: 0"><button id="theme-picker" aria-label="Pick another theme!"><img src="../brush.svg"
width="18" alt="Pick another theme!"></button>
<div id="theme-choices"></div>
</div>
<!-- Theme Picker -->
<!-- End scripts/rustdoc-before.html -->
<h1 class="title">RISC-V Ox64 BL808 SBC: NuttX Apps and Initial RAM Disk</h1>
<nav id="rustdoc"><ul>
<li><a href="#inside-a-nuttx-app" title="Inside a NuttX App">1 Inside a NuttX App</a><ul></ul></li>
<li><a href="#nuttx-app-calls-nuttx-kernel" title="NuttX App calls NuttX Kernel">2 NuttX App calls NuttX Kernel</a><ul></ul></li>
<li><a href="#nuttx-kernel-handles-system-call" title="NuttX Kernel handles System Call">3 NuttX Kernel handles System Call</a><ul></ul></li>
<li><a href="#system-call-in-action" title="System Call in Action">4 System Call in Action</a><ul></ul></li>
<li><a href="#kernel-accesses-app-memory" title="Kernel Accesses App Memory">5 Kernel Accesses App Memory</a><ul></ul></li>
<li><a href="#kernel-starts-a-nuttx-app" title="Kernel Starts a NuttX App">6 Kernel Starts a NuttX App</a><ul></ul></li>
<li><a href="#initial-ram-disk" title="Initial RAM Disk">7 Initial RAM Disk</a><ul></ul></li>
<li><a href="#mount-the-initial-ram-disk" title="Mount the Initial RAM Disk">8 Mount the Initial RAM Disk</a><ul></ul></li>
<li><a href="#pad-the-initial-ram-disk" title="Pad the Initial RAM Disk">9 Pad the Initial RAM Disk</a><ul></ul></li>
<li><a href="#whats-next" title="What’s Next">10 What’s Next</a><ul></ul></li></ul></nav><p>📝 <em>26 Nov 2023</em></p>
<p><img src="https://lupyuen.github.io/images/app-title.png" alt="NuttX App makes a System Call to NuttX Kernel" /></p>
<p>In Asia the wise folks say…</p>
<blockquote>
<p><em>“One can hide on a certain day but cannot hide for a long time”</em></p>
</blockquote>
<blockquote>
<p>“躲过初一，躲不过十五”</p>
</blockquote>
<p>In other words…</p>
<blockquote>
<p><em>“Transformers? More than meets the eye!”</em></p>
</blockquote>
<p>In this article, we go behind the shadow puppetry <em>(wayang kulit)</em> and deceptive simplicity of <strong>NuttX Applications</strong> inside <a href="https://lupyuen.github.io/articles/ox2"><strong>Apache NuttX RTOS</strong></a> (Real-Time Operating System) for <a href="https://pine64.org/documentation/Ox64/Ox64/"><strong>Pine64 Ox64 BL808</strong></a> 64-bit RISC-V SBC (pic below)…</p>
<ul>
<li>
<p>What’s inside the <strong>simplest NuttX App</strong></p>
</li>
<li>
<p>How NuttX Apps make <strong>RISC-V System Calls</strong> to NuttX Kernel</p>
</li>
<li>
<p><strong>Virtual Memory</strong> for NuttX Apps</p>
</li>
<li>
<p>Loading of <strong>ELF Executables</strong> by NuttX Kernel</p>
</li>
<li>
<p>Bundling of NuttX Apps into the <strong>Initial RAM Disk</strong></p>
</li>
<li>
<p>How we found the <strong>right spot to park</strong> our Initial RAM Disk</p>
</li>
</ul>
<p><img src="https://lupyuen.github.io/images/ox64-sbc.jpg" alt="Pine64 Ox64 64-bit RISC-V SBC (Bouffalo Lab BL808)" /></p>
<h1 id="inside-a-nuttx-app"><a class="doc-anchor" href="#inside-a-nuttx-app">§</a>1 Inside a NuttX App</h1>
<p><em>What happens inside the simplest NuttX App?</em></p>
<div class="example-wrap"><pre class="language-c"><code>// From https://github.com/apache/nuttx-apps/blob/master/examples/hello/hello_main.c#L36-L40
int main(int argc, FAR char *argv[]) {
printf(&quot;Hello, World!!\n&quot;);
return 0;
}</code></pre></div>
<p>Let’s find out! First we build <a href="https://lupyuen.github.io/articles/mmu#appendix-build-and-run-nuttx"><strong>NuttX for Ox64 BL808 SBC</strong></a>.</p>
<p>Which produces this <strong>ELF Executable</strong> for our NuttX App…</p>
<div class="example-wrap"><pre class="language-bash"><code>## ELF Executable for `hello` looks big...
$ ls -l ../apps/bin/hello
-rwxr-xr-x 518,192 ../apps/bin/hello
## Though not much inside, mostly Debug Info...
$ riscv64-unknown-elf-size ../apps/bin/hello
text data bss dec hex filename
3814 8 4 3826 ef2 ../apps/bin/hello
## Dump the RISC-V Disassembly to `hello.S`
$ riscv64-unknown-elf-objdump \
--syms --source --reloc --demangle --line-numbers --wide \
--debugging \
../apps/bin/hello \
&gt;hello.S \
2&gt;&amp;1</code></pre></div>
<p><a href="https://github.com/lupyuen2/wip-nuttx/releases/tag/ox64a-1">(See the <strong>Build Outputs</strong>)</a></p>
<p>Here’s the <strong>RISC-V Disassembly</strong> of our NuttX App: <a href="https://github.com/lupyuen2/wip-nuttx/releases/download/ox64a-1/hello.S">hello.S</a></p>
<div class="example-wrap"><pre class="language-text"><code>## Omitted: _start() prepares for signals (sig_trampoline) and calls main()
003e &lt;main&gt;:
int main(int argc, FAR char *argv[]) {
3e: 1141 addi sp,sp,-16 ## Subtract 16 from Stack Pointer
## Set Register A0 (Arg 0) to &quot;Hello, World!!\n&quot;
40: 00000517 auipc a0,0x0 40: R_RISCV_PCREL_HI20 .LC0
44: 00050513 mv a0,a0 44: R_RISCV_PCREL_LO12_I .L0
printf(&quot;Hello, World!!\n&quot;);
48: e406 sd ra,8(sp) ## Save Return Address to Stack Pointer, Offset 8
4a: 00000097 auipc ra,0x0 4a: R_RISCV_CALL puts
4e: 000080e7 jalr ra # 4a &lt;.LVL1+0x2&gt; ## Call puts()
return 0;
52: 60a2 ld ra,8(sp) ## Load Return Address from Stack Pointer, Offset 8
54: 4501 li a0,0 ## Set Return Value to 0
56: 0141 addi sp,sp,16 ## Add 16 to Stack Pointer
58: 8082 ret ## Return to caller: _start()
## Followed by the code for puts(), lib_fwrite_unlocked(), write(), ...</code></pre></div>
<p>In the RISC-V Disassembly, we see that <a href="https://github.com/apache/nuttx-apps/blob/master/examples/hello/hello_main.c#L36-L40"><strong>main</strong></a> calls…</p>
<ul>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_puts.c#L34-L96"><strong>puts</strong></a> which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_libfwrite.c#L45-L200"><strong>lib_fwrite_unlocked</strong></a> which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_libfwrite.c#L149"><strong>write</strong></a> which calls…</p>
</li>
<li>
<p><strong>NuttX Kernel</strong> to print “Hello World”</p>
</li>
</ul>
<p>How will <a href="https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_libfwrite.c#L149"><strong>write</strong></a> call the NuttX Kernel? We’ll see soon!</p>
<p><em>This code looks broken…</em></p>
<div class="example-wrap"><pre class="language-text"><code>printf(&quot;Hello, World!!\n&quot;);
## Load Register RA with Program Counter + 0x0
4a: 00000097 auipc ra, 0x0
## Call the function in Register RA: puts()
4e: 000080e7 jalr ra</code></pre></div>
<p>We break it down…</p>
<ul>
<li>
<p><a href="https://five-embeddev.com/quickref/instructions.html#-rv32--integer-register-immediate-instructions"><strong><code>auipc</code></strong></a> sets Register RA to…</p>
<div class="example-wrap"><pre class="language-c"><code>Program Counter + 0x0</code></pre></div></li>
<li>
<p><a href="https://five-embeddev.com/quickref/instructions.html#-rv32--unconditional-jumps"><strong><code>jalr</code></strong></a> jumps to the Function pointed by Register RA…</p>
<p>Which we expect to be <a href="https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_puts.c#L34-L96"><strong>puts</strong></a></p>
</li>
</ul>
<p><em>Shouldn’t <code>auipc</code> add the Offset of <code>puts</code>?</em></p>
<p>Ah that’s because we’re looking at <a href="https://en.wikipedia.org/wiki/Relocation_(computing)"><strong>Relocatable Code</strong></a>!</p>
<p>The <strong><code>auipc</code></strong> Offset will be fixed up by the <strong>NuttX ELF Loader</strong> when it loads our NuttX App for execution.</p>
<p>In our RISC-V Disassembly, the <strong>Relocation Info</strong> shows that <strong><code>0x0</code></strong> will be replaced by the Offset of <a href="https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_puts.c#L34-L96"><strong>puts</strong></a>…</p>
<div class="example-wrap"><pre class="language-text"><code>printf(&quot;Hello, World!!\n&quot;);
## Why load Register RA with Program Counter + 0x0?
## Gotcha! 0x0 will be changed to the Offset of puts()
4a: 00000097 auipc ra, 0x0
4a: R_RISCV_CALL puts
## Call the function in Register RA: puts()
## Which will work when ELF Loader fixes the Offset of puts()
4e: 000080e7 jalr ra # 4a &lt;.LVL1+0x2&gt;</code></pre></div>
<p>Therefore we’re all good! (Eventually)</p>
<p><em>Why <code>puts</code> instead of <code>printf</code>?</em></p>
<p>The GCC Compiler has cleverly optimised away <strong>printf</strong> to become <strong>puts</strong>.</p>
<p>If we do this (and foil the GCC Compiler)…</p>
<div class="example-wrap"><pre class="language-c"><code>// Nope, GCC Compiler won&#39;t change printf() to puts()
printf(
&quot;Hello, World %s!!\n&quot;, // Meaningful Format String
&quot;Luppy&quot; // Makes it complicated
);</code></pre></div>
<p>Then <strong>printf</strong> will appear in our RISC-V Disassembly.</p>
<p>We circle back to <strong>write</strong>…</p>
<p><img src="https://lupyuen.github.io/images/app-syscall.jpg" alt="NuttX App calls NuttX Kernel" /></p>
<h1 id="nuttx-app-calls-nuttx-kernel"><a class="doc-anchor" href="#nuttx-app-calls-nuttx-kernel">§</a>2 NuttX App calls NuttX Kernel</h1>
<p><em>Our app will print something to the console…</em></p>
<p><em>But NuttX Apps can’t write directly to the Serial Device right?</em></p>
<p>Nope!</p>
<ul>
<li>
<p>NuttX Apps run in <strong>RISC-V User Mode</strong>…</p>
</li>
<li>
<p>Which <strong>can’t access</strong> the Serial Device (and other resources) controlled by NuttX Kernel…</p>
</li>
<li>
<p>Which runs in <strong>RISC-V Supervisor Mode</strong></p>
</li>
</ul>
<p>That’s why “<strong>write</strong>” should trigger a <strong>System Call</strong> to the NuttX Kernel, jumping from RISC-V <strong>User Mode to Supervisor Mode</strong>.</p>
<p>(And write to the Serial Device, pic above)</p>
<p><em>Will NuttX Apps need Special Coding to make System Calls?</em></p>
<p>Not at all! The System Call is <strong>totally transparent</strong> to our app…</p>
<ul>
<li>
<p>Our <strong>NuttX App</strong> will call a normal function named “<strong>write</strong>”…</p>
</li>
<li>
<p>That pretends to be the actual “<strong>write</strong>” function in <strong>NuttX Kernel</strong>…</p>
</li>
<li>
<p>By forwarding the “<strong>write</strong>” function call (and parameters)…</p>
</li>
<li>
<p>Through a <strong>RISC-V System Call</strong></p>
</li>
</ul>
<p><em>What’s this “forwarding” to a System Call?</em></p>
<p>This forwarding happens inside a <strong>Proxy Function</strong> that’s auto-generated during NuttX Build…</p>
<div class="example-wrap"><pre class="language-c"><code>// From nuttx/syscall/proxies/PROXY_write.c
// Auto-Generated Proxy for `write`
// Looks like the Kernel `write`, though it&#39;s actually a System Call
ssize_t write(int parm1, FAR const void * parm2, size_t parm3) {
// Make a System Call with 3 parameters...
return (ssize_t) sys_call3(
(unsigned int) SYS_write, // System Call Number (63 = `write`)
(uintptr_t) parm1, // File Descriptor (1 = Standard Output)
(uintptr_t) parm2, // Buffer to be written
(uintptr_t) parm3 // Number of bytes to write
);
}</code></pre></div>
<p>Our NuttX App (implicitly) calls this <strong>Proxy Version</strong> of “<strong>write</strong>” (that pretends to be the Kernel “<strong>write</strong>”)…</p>
<div class="example-wrap"><pre class="language-c"><code>// Our App calls the Proxy Function...
int ret = write(
1, // File Descriptor (1 = Standard Output)
&quot;Hello, World!!\n&quot;, // Buffer to be written
15 // Number of bytes to write
);</code></pre></div>
<p>Which triggers a <strong>System Call</strong> to the Kernel.</p>
<p>(Indeed “More than meets the eye!”)</p>
<p><em>What’s sys_call3?</em></p>
<p>It makes a <strong>System Call</strong> (to NuttX Kernel) with <strong>3 Parameters</strong>: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/include/syscall.h#L240-L268">syscall.h</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Make a System Call with 3 parameters
uintptr_t sys_call3(
unsigned int nbr, // System Call Number (63 = `write`)
uintptr_t parm1, // First Parameter
uintptr_t parm2, // Second Parameter
uintptr_t parm3 // Third Parameter
) {
// Pass the Function Number and Parameters in
// Registers A0 to A3
register long r0 asm(&quot;a0&quot;) = (long)(nbr);
register long r1 asm(&quot;a1&quot;) = (long)(parm1);
register long r2 asm(&quot;a2&quot;) = (long)(parm2);
register long r3 asm(&quot;a3&quot;) = (long)(parm3);
// `ecall` will jump from RISC-V User Mode
// to RISC-V Supervisor Mode
// to execute the System Call.
// Input + Output Registers: A0 to A3
// Clobbers the Memory
asm volatile
(
&quot;ecall&quot;
:: &quot;r&quot;(r0), &quot;r&quot;(r1), &quot;r&quot;(r2), &quot;r&quot;(r3)
: &quot;memory&quot;
);
// No-operation, does nothing
asm volatile(&quot;nop&quot; : &quot;=r&quot;(r0));
// Return the result from Register A0
return r0;
}</code></pre></div>
<p><a href="https://five-embeddev.com/quickref/instructions.html#-rv32--rv32"><strong><code>ecall</code></strong></a> is the RISC-V Instruction that jumps from RISC-V <strong>User Mode to Supervisor Mode</strong>…</p>
<p>That allows NuttX Kernel to execute the actual “<strong>write</strong>” function, with the real Serial Device.</p>
<p>(We’ll explain how)</p>
<p><em>Why the no-op after ecall?</em></p>
<p>We’re guessing: It might be reserved for special calls to NuttX Kernel in future.</p>
<p><a href="https://lupyuen.github.io/articles/semihost#decipher-the-risc-v-exception">(Similar to <strong><code>ebreak</code></strong> for Semihosting)</a></p>
<p><em>Every System Call to NuttX Kernel has its own Proxy Function?</em></p>
<p>Yep! We can see the Auto-Generated <strong>Proxy Functions</strong> for each System Call…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Proxy Functions called by `hello` app
$ grep PROXY hello.S
PROXY__assert.c
PROXY__exit.c
PROXY_clock_gettime.c
PROXY_gettid.c
PROXY_lseek.c
PROXY_nxsem_wait.c
PROXY_sem_clockwait.c
PROXY_sem_destroy.c
PROXY_sem_post.c
PROXY_sem_trywait.c
PROXY_task_setcancelstate.c
PROXY_write.c</code></pre></div>
<p>Next we figure out how System Calls will work…</p>
<p><img src="https://lupyuen.github.io/images/app-syscall2.jpg" alt="NuttX Kernel handles System Call" /></p>
<h1 id="nuttx-kernel-handles-system-call"><a class="doc-anchor" href="#nuttx-kernel-handles-system-call">§</a>3 NuttX Kernel handles System Call</h1>
<p><em>Our App makes an ecall to jump to NuttX Kernel (pic above)…</em></p>
<p><em>What happens on the other side?</em></p>
<p>Remember the Proxy Function from earlier? Now we do the exact opposite in our <strong>Stub Function</strong> (that runs in the Kernel)…</p>
<div class="example-wrap"><pre class="language-c"><code>// From nuttx/syscall/stubs/STUB_write.c
// Auto-Generated Stub File for `write`
// This runs in NuttX Kernel triggered by `ecall`.
// We make the actual call to `write`.
// (`nbr` is Offset in Stub Lookup Table, unused)
uintptr_t STUB_write(int nbr, uintptr_t parm1, uintptr_t parm2, uintptr_t parm3) {
// Call the Kernel version of `write`
return (uintptr_t) write(
(int) parm1, // File Descriptor (1 = Standard Output)
(FAR const void *) parm2, // Buffer to be written
(size_t) parm3 // Number of bytes to write
); // Return the result to the App
}</code></pre></div>
<p>Thus our <strong>NuttX Build</strong> auto-generates 2 things…</p>
<ul>
<li>
<p><strong>Proxy Function</strong> (runs in NuttX Apps)</p>
</li>
<li>
<p><strong>Stub Function</strong> (runs in NuttX Kernel)</p>
</li>
</ul>
<p>This happens for <strong>every System Call</strong> exposed by NuttX Kernel…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Stub Functions in NuttX Kernel
$ grep STUB nuttx.S
STUB__assert.c
STUB__exit.c
STUB_boardctl.c
STUB_chmod.c
STUB_chown.c
...</code></pre></div>
<p><a href="https://nuttx.apache.org/docs/latest/components/syscall.html">(More about <strong>Proxy and Stub Functions</strong>)</a></p>
<p><em>Who calls STUB_write?</em></p>
<p>When our NuttX App makes an <strong><code>ecall</code></strong>, it triggers <strong>IRQ 8</strong> <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/include/irq.h#L52-L75"><strong>(RISCV_IRQ_ECALLU)</strong></a> that’s <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_exception.c#L114-L119"><strong>handled by</strong></a>…</p>
<ul>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_swint.c#L105-L537"><strong>riscv_swint</strong></a> which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_swint.c#L54-L100"><strong>dispatch_syscall</strong></a> which calls the Kernel Stub Function (<strong>STUB_write</strong>) and…</p>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/supervisor/riscv_syscall.S#L49-L177"><strong>sys_call2</strong></a> with A0 set to <strong>SYS_syscall_return</strong> (3) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/supervisor/riscv_perform_syscall.c#L36-L78"><strong>riscv_perform_syscall</strong></a> which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_swint.c#L105-L537"><strong>riscv_swint</strong></a> with IRQ 0, to return from the <strong><code>ecall</code></strong></p>
</li>
</ul>
<p><em>How will dispatch_syscall know which Stub Function to call?</em></p>
<p>Remember that our Proxy Function (in NuttX App) passes the <strong>System Call Number</strong> for “<strong>write</strong>”?</p>
<div class="example-wrap"><pre class="language-c"><code>// From nuttx/syscall/proxies/PROXY_write.c
// Auto-Generated Proxy for `write`, called by NuttX App
ssize_t write(int parm1, FAR const void * parm2, size_t parm3) {
// Make a System Call with 3 parameters...
return (ssize_t) sys_call3(
(unsigned int) SYS_write, // System Call Number (63 = `write`)
...</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_swint.c#L54-L100"><strong>dispatch_syscall</strong></a> (in NuttX Kernel) will look up the System Call Number in the <a href="https://github.com/apache/nuttx/blob/master/syscall/syscall_stublookup.c#L80-L93"><strong>Stub Lookup Table</strong></a>. And fetch the <strong>Stub Function</strong> to call.</p>
<p><em>How did we figure out that 63 is the System Call Number for “write”?</em></p>
<p>OK this gets tricky. Below is the Enum that defines all <strong>System Call Numbers</strong>: <a href="https://github.com/apache/nuttx/blob/master/include/sys/syscall.h#L55-L66">syscall.h</a> and <a href="https://github.com/apache/nuttx/blob/master/include/sys/syscall_lookup.h#L202">syscall_lookup.h</a></p>
<div class="example-wrap"><pre class="language-c"><code>// System Call Enum sequentially assigns
// all System Call Numbers (8 to 147-ish)
enum {
...
SYSCALL_LOOKUP(close, 1) // 1 Parameter
SYSCALL_LOOKUP(ioctl, 3) // 3 Parameters
SYSCALL_LOOKUP(read, 3) // 3 Parameters
SYSCALL_LOOKUP(write, 3) // 3 Parameters
...
};</code></pre></div>
<p>However it’s an Enum, <strong>numbered sequentially</strong> from 8 to 147-ish. We won’t literally see 63 in the NuttX Source Code.</p>
<p>Then we lookup the <strong>Debug Info</strong> in the RISC-V Disassembly for NuttX Kernel: <a href="https://github.com/lupyuen2/wip-nuttx/releases/download/ox64a-1/nuttx.S">nuttx.S</a></p>
<div class="example-wrap"><pre class="language-text"><code>Abbrev Number: 6 (DW_TAG_enumerator)
DW_AT_name : SYS_write
DW_AT_const_value : 63</code></pre></div>
<p>Whoomp there it is! Says here that “<strong>write</strong>” is <strong>System Call #63</strong>.</p>
<p><em>That’s an odd way to define System Call Numbers…</em></p>
<p>Yeah it’s <strong>not strictly an immutable ABI</strong> like Linux, because our System Call Numbers may change! It depends on the <a href="https://github.com/apache/nuttx/blob/master/include/sys/syscall_lookup.h#L90-L152"><strong>Build Options</strong></a> that we select.</p>
<p><a href="https://en.wikipedia.org/wiki/Application_binary_interface">(ABI means <strong>Application Binary Interface</strong>)</a></p>
<p>Though there’s a jolly good thing: It’s super simple to experiment with <strong>new System Calls</strong>!</p>
<p><a href="https://github.com/apache/nuttx/blob/master/syscall/syscall.csv#L209-L210">(Just add to <strong>NuttX System Calls</strong>)</a></p>
<p><a href="https://nuttx.apache.org/docs/latest/components/syscall.html">(As explained here)</a></p>
<p><img src="https://lupyuen.github.io/images/app-title.png" alt="NuttX App calls NuttX Kernel" /></p>
<h1 id="system-call-in-action"><a class="doc-anchor" href="#system-call-in-action">§</a>4 System Call in Action</h1>
<p><em>This looks complicated… It works right?</em></p>
<p>Yep we have solid evidence, from <a href="https://lupyuen.github.io/articles/mmu#appendix-build-and-run-nuttx"><strong>NuttX for Ox64 BL808 SBC</strong></a>!</p>
<p>Remember to enable <strong>System Call Logging</strong> in “<code>make menuconfig</code>”…</p>
<div class="example-wrap"><pre class="language-text"><code>Build Setup
&gt; Debug Options
&gt; Syscall Debug Features
&gt; Enable &quot;Syscall Warning, Error and Info&quot;</code></pre></div>
<p>Watch what happens when we <strong>boot NuttX on Ox64</strong> (pic above)…</p>
<ul>
<li>
<p>Our app (NuttX Shell) begins by <strong>printing something</strong> to the console.</p>
</li>
<li>
<p>It makes an <strong><code>ecall</code></strong> for System Call #63 “<strong>write</strong>”.</p>
</li>
<li>
<p>Which triggers <strong>IRQ 8</strong> and jumps to <strong>NuttX Kernel</strong></p>
</li>
</ul>
<div class="example-wrap"><pre class="language-text"><code>riscv_dispatch_irq: irq=8
riscv_swint: Entry: regs: 0x5040bcb0 cmd: 63
EPC: 800019b2
A0: 003f A1: 0001 A2: 8000ad00 A3: 001e</code></pre></div>
<p><a href="https://gist.github.com/lupyuen/ce82b29c664b1d5898b6a59743310c17#file-ox64-nuttx-ecall-log-L563-L588">(See the <strong>Complete Log</strong>)</a></p>
<p>The <strong>RISC-V Registers</strong> look familiar…</p>
<ul>
<li>
<p>A0 is <strong><code>0x3F</code></strong></p>
<p>(System Call #63 for “<strong>write</strong>”)</p>
</li>
<li>
<p>A1 is <strong><code>1</code></strong></p>
<p>(File Descriptor #1 for Standard Output)</p>
</li>
<li>
<p>A2 is <strong><code>0x8000_AD00</code></strong></p>
<p>(Buffer to be written)</p>
</li>
<li>
<p>A3 is <strong><code>0x1E</code></strong></p>
<p>(Number of bytes to write)</p>
</li>
</ul>
<p>NuttX Kernel calls our Stub Function <strong>STUB_write</strong>…</p>
<div class="example-wrap"><pre class="language-text"><code>riscv_swint: SWInt Return: 37
STUB_write: nbr=440, parm1=1, parm2=8000ad00, parm3=1e
NuttShell (NSH) NuttX-12.0.3</code></pre></div>
<p>Which calls Kernel “<strong>write</strong>” and <strong>prints the text</strong>: “NuttShell”</p>
<p>Then NuttX Kernel completes the <strong><code>ecall</code></strong>…</p>
<div class="example-wrap"><pre class="language-text"><code>riscv_swint: Entry: regs: 0x5040baa0 cmd: 3
EPC: 80001a6a
A0: 0003 A1: 5040bbec A2: 001e A3: 0000
riscv_swint: SWInt Return: 1e</code></pre></div>
<ul>
<li>
<p>A0 is <strong>3</strong></p>
<p>(Return from System Call: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/include/syscall.h#L80-L87"><strong>SYS_syscall_return</strong></a>)</p>
</li>
<li>
<p>A2 is <strong><code>0x1E</code></strong></p>
<p>(Number of bytes written)</p>
</li>
</ul>
<p>And returns the result <strong><code>0x1E</code></strong> to our NuttX App. <a href="https://five-embeddev.com/quickref/instructions.html#-supervisor--sstatus">(Via <strong><code>sret</code></strong>)</a></p>
<p>Our NuttX App has successfully made a System Call on Ox64 yay!</p>
<p><a href="https://gist.github.com/lupyuen/ce82b29c664b1d5898b6a59743310c17#file-ox64-nuttx-ecall-log-L563-L588">(See the <strong>Complete Log</strong>)</a></p>
<p><img src="https://lupyuen.github.io/images/mmu-l3user.jpg" alt="Virtual Memory for NuttX App" /></p>
<p><a href="https://lupyuen.github.io/articles/mmu#virtual-memory"><em>Virtual Memory for NuttX App</em></a></p>
<h1 id="kernel-accesses-app-memory"><a class="doc-anchor" href="#kernel-accesses-app-memory">§</a>5 Kernel Accesses App Memory</h1>
<p><em>NuttX Kernel prints the buffer at <code>0x8000_AD00</code>…</em></p>
<p><em>It doesn’t look like a RAM Address?</em></p>
<p>That’s a <strong>Virtual Memory Address</strong>…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/mmu"><strong>“RISC-V Ox64 BL808 SBC: Sv39 Memory Management Unit”</strong></a></li>
</ul>
<p>TLDR? No worries…</p>
<ul>
<li>
<p><strong>Kernel RAM</strong> is at <strong><code>0x5000_0000</code></strong></p>
</li>
<li>
<p>Which gets dished out dynamically to <strong>NuttX Apps</strong></p>
</li>
<li>
<p>And becomes <strong>Virtual Memory</strong> at <strong><code>0x8000_0000</code></strong> (pic above)</p>
</li>
</ul>
<p>Hence our NuttX App has passed a chunk of its own <strong>Virtual Memory</strong>. And NuttX Kernel happily prints it!</p>
<p><em>Huh? NuttX Kernel can access Virtual Memory?</em></p>
<ol>
<li>
<p>NuttX uses 2 sets of Page Tables: <strong>Kernel Page Table</strong> and <strong>User Page Table</strong>.</p>
<p>(User Page Table defines the <strong>Virtual Memory</strong> for NuttX Apps)</p>
</li>
<li>
<p>According to the <a href="https://gist.github.com/lupyuen/ce82b29c664b1d5898b6a59743310c17#file-ox64-nuttx-ecall-log-L321-L323"><strong>NuttX Log</strong></a>, the Kernel swaps the <a href="https://lupyuen.github.io/articles/mmu#swap-the-satp-register"><strong>RISC-V SATP Register</strong></a> from Kernel Page Table to <strong>User Page Table</strong>…</p>
<p>And doesn’t swap back!</p>
</li>
<li>
<p>Which means the <strong>User Page Table</strong> is still in effect!</p>
<p>And the <strong>Virtual Memory</strong> at <strong><code>0x8000_0000</code></strong> is perfectly accessible by the Kernel.</p>
</li>
<li>
<p>There’s a catch: <strong>RISC-V Supervisor Mode</strong> (NuttX Kernel) may access the Virtual Memory mapped to <strong>RISC-V User Mode</strong> (NuttX Apps)…</p>
<p>Only if the <a href="https://five-embeddev.com/riscv-isa-manual/latest/supervisor.html#sec:translation"><strong>SUM Bit is set in SSTATUS Register</strong></a>!</p>
<p><a href="https://five-embeddev.com/riscv-isa-manual/latest/supervisor.html#sstatus">(SUM Bit will permit <strong>Supervisor User Memory</strong> access)</a></p>
</li>
<li>
<p>And that’s absolutely hunky dory because at NuttX Startup, <a href="https://github.com/apache/nuttx/blob/master/sched/init/nx_start.c#L298-L713"><strong>nx_start</strong></a> calls…</p>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_initialstate.c#L41-L140"><strong>up_initial_state</strong></a> which calls…</p>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_getnewintctx.c#L74-L81"><strong>riscv_set_idleintctx</strong></a> to set the <strong>SUM Bit</strong> in SSTATUS Register</p>
<p><a href="https://lupyuen.github.io/articles/ox2#appendix-nuttx-boot-flow">(How NuttX calls <strong>nx_start</strong>)</a></p>
</li>
</ol>
<p>That’s why NuttX Kernel can access Virtual Memory (passed by NuttX Apps) at <strong><code>0x8000_0000</code></strong>!</p>
<p><img src="https://lupyuen.github.io/images/app-flow.jpg" alt="Kernel Starts a NuttX App" /></p>
<p><a href="https://github.com/lupyuen/nuttx-ox64#kernel-starts-a-nuttx-app"><em>Clickable Version of NuttX Flow</em></a></p>
<h1 id="kernel-starts-a-nuttx-app"><a class="doc-anchor" href="#kernel-starts-a-nuttx-app">§</a>6 Kernel Starts a NuttX App</h1>
<p><em>Alrighty NuttX Apps can call NuttX Kernel…</em></p>
<p><em>But how does NuttX Kernel start a NuttX App?</em></p>
<p>Previously we walked through the <strong>Boot Sequence</strong> for NuttX…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/ox2#appendix-nuttx-boot-flow"><strong>“NuttX Boot Flow”</strong></a></li>
</ul>
<p>Right after that, <a href="https://github.com/apache/nuttx/blob/master/sched/init/nx_bringup.c#L373-L458"><strong>NuttX Bringup (nx_bringup)</strong></a> calls (pic above)…</p>
<ul>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/sched/init/nx_bringup.c#L330-L367"><strong>Create Init Thread: nx_create_initthread</strong></a> (to create the Init Thread) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/sched/init/nx_bringup.c#L212C1-L302"><strong>Start App: nx_start_application</strong></a> (to start NuttX Shell) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/binfmt/binfmt_exec.c#L183-L223"><strong>Exec Spawn: exec_spawn</strong></a> (to start the app) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/binfmt/binfmt_exec.c#L42-L179"><strong>Exec Internal: exec_internal</strong></a> (to start the app) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/binfmt/binfmt_loadmodule.c#L136-L225"><strong>Load Module: load_module</strong></a> (to load the app, see below) and…</p>
<p><a href="https://github.com/apache/nuttx/blob/master/binfmt/binfmt_execmodule.c#L190-L450"><strong>Execute Module: exec_module</strong></a> (to execute the app)</p>
</li>
</ul>
<p>To load a NuttX App module: <a href="https://github.com/apache/nuttx/blob/master/binfmt/binfmt_loadmodule.c#L136-L225"><strong>load_module</strong></a> calls…</p>
<ul>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/binfmt/binfmt_loadmodule.c#L83-L132"><strong>Load Absolute Module: load_absmodule</strong></a> (to load an absolute path) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/include/nuttx/binfmt/binfmt.h#L122-L148"><strong>Load Binary Format: binfmt_s.load</strong></a> (to load a binary module) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/binfmt/elf.c#L84-L94"><strong>ELF Loader: g_elfbinfmt</strong></a> (to load the ELF File, see below)</p>
</li>
</ul>
<p>To load the ELF File: <a href="https://github.com/apache/nuttx/blob/master/binfmt/elf.c#L84-L94"><strong>ELF Loader g_elfbinfmt</strong></a> calls…</p>
<ul>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/binfmt/elf.c#L225-L355"><strong>Load ELF Binary: elf_loadbinary</strong></a> (to load the ELF Binary) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/binfmt/libelf/libelf_load.c#L297-L445"><strong>Load ELF: elf_load</strong></a> (to load the ELF Binary) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/binfmt/libelf/libelf_addrenv.c#L56-L178"><strong>Allocate Address Env: elf_addrenv_alloc</strong></a> (to allocate the Address Env) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_addrenv.c#L339-L490"><strong>Create Address Env: up_addrenv_create</strong></a> (to create the Address Env) which calls…</p>
<p>(Also calls <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_mmu.h#L152-L176"><strong>mmu_satp_reg</strong></a> to set SATP Register)</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_addrenv.c#L213-L310"><strong>Create MMU Region: create_region</strong></a> (to create the MMU Region) which calls…</p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/common/riscv_mmu.c#L62-L109"><strong>Set MMU Page Table Entry: mmu_ln_setentry</strong></a> (to populate the Page Table Entries)</p>
</li>
</ul>
<p>There’s plenty happening inside <a href="https://github.com/apache/nuttx/blob/master/binfmt/binfmt_execmodule.c#L190-L450"><strong>Execute Module: exec_module</strong></a>. Too bad we won’t explore today.</p>
<p><a href="https://github.com/lupyuen/nuttx-ox64#kernel-starts-a-nuttx-app">(<strong>Clickable Version</strong> of NuttX Flow)</a></p>
<p><img src="https://lupyuen.github.io/images/semihost-title.jpg" alt="Initial RAM Disk for Star64 JH7110" /></p>
<p><a href="https://lupyuen.github.io/articles/semihost"><em>Initial RAM Disk for Star64 JH7110</em></a></p>
<h1 id="initial-ram-disk"><a class="doc-anchor" href="#initial-ram-disk">§</a>7 Initial RAM Disk</h1>
<p><em>OK we know how NuttX Kernel starts a NuttX App…</em></p>
<p><em>But where are the NuttX Apps stored?</em></p>
<p>Right now we’re working with the <strong>Early Port of NuttX</strong> to Ox64 BL808 SBC. We can’t access the File System in the microSD Card.</p>
<p>All we have: A File System that <strong>lives in RAM</strong> and contains our <strong>NuttX Shell + NuttX Apps</strong>.</p>
<p>That’s our <strong>Initial RAM Disk: initrd</strong></p>
<div class="example-wrap"><pre class="language-bash"><code>## Build the Apps Filesystem
make -j 8 export
pushd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make -j 8 import
popd
## Generate the Initial RAM Disk `initrd`
## in ROMFS Filesystem Format
## from the Apps Filesystem `../apps/bin`
## and label it `NuttXBootVol`
genromfs \
-f initrd \
-d ../apps/bin \
-V &quot;NuttXBootVol&quot;</code></pre></div>
<p><a href="https://lupyuen.github.io/articles/romfs#inside-a-rom-fs-filesystem">(Inside a <strong>ROM FS Filesystem</strong>)</a></p>
<p><em>How to load the Initial RAM Disk from microSD to RAM?</em></p>
<p><a href="https://lupyuen.github.io/articles/linux#u-boot-bootloader-for-star64"><strong>U-Boot Bootloader</strong></a> will do it for us!</p>
<p>Two ways that U-Boot can load the Initial RAM Disk from microSD…</p>
<ol>
<li>
<p>Load the Initial RAM Disk from a <strong>Separate File: initrd</strong> (similar to Star64, pic above)</p>
<p>This means we modify the <a href="https://github.com/openbouffalo/buildroot_bouffalo/blob/main/board/pine64/ox64/boot-pine64.cmd"><strong>U-Boot Script: boot-pine64.scr</strong></a></p>
<p>And make it <a href="https://lupyuen.github.io/articles/semihost#appendix-boot-nuttx-over-tftp-with-initial-ram-disk"><strong>load the initrd</strong></a> file into RAM.</p>
<p>(Which is good for separating the NuttX Kernel and NuttX Apps)</p>
<p>OR…</p>
</li>
<li>
<p>Append the Initial RAM Disk to the <strong>NuttX Kernel Image</strong></p>
<p>U-Boot Bootloader will load (one-shot into RAM) the NuttX Kernel + Initial RAM Disk.</p>
<p>And we reuse the existing <strong>U-Boot Config</strong> on the microSD Card: <a href="https://github.com/openbouffalo/buildroot_bouffalo/blob/main/board/pine64/ox64/rootfs-overlay/boot/extlinux/extlinux.conf"><strong>extlinux/extlinux.conf</strong></a></p>
<p>(Which might be more efficient for our Limited RAM)</p>
<p><a href="https://github.com/openbouffalo/buildroot_bouffalo/wiki/U-Boot-Bootflow">(More about the <strong>U-Boot Boot Flow</strong>)</a></p>
</li>
</ol>
<p>Since Ox64 is low on RAM, we’ll do the <strong>Second Method</strong> (Append to Kernel). Like this…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Export the NuttX Kernel to `nuttx.bin`
riscv64-unknown-elf-objcopy \
-O binary \
nuttx \
nuttx.bin
## Prepare a Padding with 64 KB of zeroes
head -c 65536 /dev/zero &gt;/tmp/nuttx.pad
## Append Padding and Initial RAM Disk to NuttX Kernel
cat nuttx.bin /tmp/nuttx.pad initrd \
&gt;Image
## Overwrite the Linux Image on Ox64 microSD
cp Image &quot;/Volumes/NO NAME/&quot;
## U-Boot Bootloader will load NuttX Kernel and
## Initial RAM Disk into RAM</code></pre></div>
<p>This is how we made it work…</p>
<p><a href="https://github.com/lupyuen/lupyuen.github.io/issues/24">(Ox64 can boot NuttX from <strong>Flash Memory</strong>)</a></p>
<p><img src="https://lupyuen.github.io/images/app-initrd.jpg" alt="Initial RAM Disk for Ox64" /></p>
<h1 id="mount-the-initial-ram-disk"><a class="doc-anchor" href="#mount-the-initial-ram-disk">§</a>8 Mount the Initial RAM Disk</h1>
<p><em>We appended the Initial RAM Disk to NuttX Kernel (pic above)…</em></p>
<p><em>U-Boot Bootloader loads the NuttX Kernel + Initial RAM Disk into RAM…</em></p>
<p><em>How in RAM will NuttX Kernel locate the Initial RAM Disk?</em></p>
<p>Our Initial RAM Disk follows the <a href="https://docs.kernel.org/filesystems/romfs.html"><strong>ROM File System Format</strong></a> (ROM FS). We <strong>search our RAM</strong> for the ROM File System by its Magic Number.</p>
<p>Then we copy it into the designated <strong>Memory Region</strong> for mounting: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_start.c#L104-L177">bl808_start.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Locate the Initial RAM Disk and copy to the designated Memory Region
void bl808_copy_ramdisk(void) {
// After _edata, search for &quot;-rom1fs-&quot;. This is the RAM Disk Address.
// Limit search to 256 KB after Idle Stack Top.
const char *header = &quot;-rom1fs-&quot;;
uint8_t *ramdisk_addr = NULL;
for (uint8_t *addr = _edata; addr &lt; (uint8_t *)BL808_IDLESTACK_TOP + (256 * 1024); addr++) {
if (memcmp(addr, header, strlen(header)) == 0) {
ramdisk_addr = addr;
break;
}
}
// Stop if RAM Disk is missing
if (ramdisk_addr == NULL) { PANIC(); }
// RAM Disk must be after Idle Stack, to prevent overwriting
if (ramdisk_addr &lt;= (uint8_t *)BL808_IDLESTACK_TOP) { PANIC(); }
// Read the Filesystem Size from the next 4 bytes, in Big Endian
// Add 0x1F0 to Filesystem Size
const uint32_t size =
(ramdisk_addr[8] &lt;&lt; 24) +
(ramdisk_addr[9] &lt;&lt; 16) +
(ramdisk_addr[10] &lt;&lt; 8) +
ramdisk_addr[11] +
0x1F0;
// Filesystem Size must be less than RAM Disk Memory Region
if (size &gt; (size_t)__ramdisk_size) { PANIC(); }
// Copy the Filesystem bytes to RAM Disk Memory Region
// Warning: __ramdisk_start overlaps with ramdisk_addr + size
// Which doesn&#39;t work with memcpy.
// Sadly memmove is aliased to memcpy, so we implement memmove ourselves
bl808_copy_overlap((void *)__ramdisk_start, ramdisk_addr, size);
}</code></pre></div>
<p>(More about <strong>edata</strong>, <strong>Idle Stack</strong> and <strong>bl808_copy_overlap</strong> in the next section)</p>
<p><em>Why did we copy Initial RAM Disk to ramdisk_start?</em></p>
<p><strong>ramdisk_start</strong> points to the Memory Region that we reserved for mounting our RAM Disk.</p>
<p>It’s defined in the <strong>NuttX Linker Script</strong>: <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/bl808/ox64/scripts/ld.script#L21-L48">ld.script</a></p>
<div class="example-wrap"><pre class="language-text"><code>/* Memory Region for Mounting RAM Disk */
ramdisk (rwx) : ORIGIN = 0x50A00000, LENGTH = 16M
...
__ramdisk_start = ORIGIN(ramdisk);
__ramdisk_size = LENGTH(ramdisk);
__ramdisk_end = ORIGIN(ramdisk) + LENGTH(ramdisk);</code></pre></div>
<p><em>Who calls the code above?</em></p>
<p>We locate and copy the Initial RAM Disk at the very top of our <strong>NuttX Start Code</strong>.</p>
<p>This just after <strong>erasing the BSS</strong> (Global and Static Variables), in case we need to print some messages and it uses Global and Static Variables: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_start.c#L254-L284">bl808_start.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// NuttX Start Code
void bl808_start(int mhartid) {
// Clear the BSS for Global and Static Variables
bl808_clear_bss();
// Copy the RAM Disk
bl808_copy_ramdisk();</code></pre></div>
<p>Later during startup, we <strong>mount the RAM Disk</strong> from the Memory Region: <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/bl808/ox64/src/bl808_appinit.c#L51-L87">bl808_appinit.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// After NuttX has booted...
void board_late_initialize(void) {
// Mount the RAM Disk
mount_ramdisk();
}
// Mount the RAM Disk
int mount_ramdisk(void) {
desc.minor = RAMDISK_DEVICE_MINOR;
desc.nsectors = NSECTORS((ssize_t)__ramdisk_size);
desc.sectsize = SECTORSIZE;
desc.image = __ramdisk_start;
ret = boardctl(BOARDIOC_ROMDISK, (uintptr_t)&amp;desc);</code></pre></div>
<p><a href="https://lupyuen.github.io/articles/ox2#appendix-nuttx-boot-flow">(How NuttX calls <strong>board_late_initialize</strong>)</a></p>
<p>All this works great: NuttX mounts our RAM Disk successfully, and starts the ELF Executable for NuttX Shell!</p>
<div class="example-wrap"><pre class="language-text"><code>bl808_copy_ramdisk:
_edata=0x50400258, _sbss=0x50400290, _ebss=0x50407000, BL808_IDLESTACK_TOP=0x50407c00
ramdisk_addr=0x50408288
size=8192016
Before Copy: ramdisk_addr=0x50408288
After Copy: __ramdisk_start=0x50a00000
...
elf_initialize: Registering ELF
uart_register: Registering /dev/console
work_start_lowpri: Starting low-priority kernel worker thread(s)
nx_start_application: Starting init task: /system/bin/init
load_absmodule: Loading /system/bin/init
elf_loadbinary: Loading file: /system/bin/init
elf_init: filename: /system/bin/init loadinfo: 0x5040c618
elf_read: Read 64 bytes from offset 0</code></pre></div>
<p>(“<strong>system/bin/init</strong>” is the NuttX Shell)</p>
<p><a href="https://gist.github.com/lupyuen/74a44a3e432e159c62cc2df6a726cb89#file-ox64-nuttx13-log-L114-L159">(See the <strong>Complete Log</strong>)</a></p>
<p>Last thing for today: The mysterious 64 KB padding…</p>
<p><img src="https://lupyuen.github.io/images/app-initrd2.jpg" alt="Initial RAM Disk for Ox64" /></p>
<h1 id="pad-the-initial-ram-disk"><a class="doc-anchor" href="#pad-the-initial-ram-disk">§</a>9 Pad the Initial RAM Disk</h1>
<p><em>Between NuttX Kernel and Initial RAM Disk…</em></p>
<p><em>Why did we pad 64 KB of zeroes? (Pic above)</em></p>
<div class="example-wrap"><pre class="language-bash"><code>## Prepare a Padding with 64 KB of zeroes
head -c 65536 /dev/zero &gt;/tmp/nuttx.pad
## Append Padding and Initial RAM Disk to NuttX Kernel
cat nuttx.bin /tmp/nuttx.pad initrd \
&gt;Image
## U-Boot Bootloader will load NuttX Kernel and
## Initial RAM Disk into RAM</code></pre></div>
<p>U-Boot Bootloader will load our Initial RAM Disk into RAM. However it’s dangerously close to <strong>BSS Memory</strong> (Global and Static Variables) and <strong>Kernel Stack</strong>.</p>
<p>There’s a risk that our Initial RAM Disk will be <strong>contaminated by BSS and Stack</strong>. This is how we found a clean, safe space for our Initial RAM Disk (pic above)…</p>
<p>We inspect the <a href="https://gist.github.com/lupyuen/74a44a3e432e159c62cc2df6a726cb89#file-ox64-nuttx13-log-L114-L118"><strong>NuttX Log</strong></a> and the <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/bl808/ox64/scripts/ld.script#L20-L28"><strong>NuttX Linker Script</strong></a>…</p>
<div class="example-wrap"><pre class="language-text"><code>// End of Data Section
_edata=0x50400258
// Start of BSS Section
_sbss=0x50400290
// End of BSS Section
_ebss=0x50407000
// Top of Kernel Idle Stack
BL808_IDLESTACK_TOP=0x50407c00
// We located the initrd after the Top of Idle Stack
ramdisk_addr=0x50408288, size=8192016
// And we copied initrd to the Memory Region for the RAM Disk
__ramdisk_start=0x50a00000</code></pre></div>
<p>Or graphically…</p>
<div><table><thead><tr><th style="text-align: left">Memory Region</th><th style="text-align: center">Start</th><th style="text-align: center">End</th></tr></thead><tbody>
<tr><td style="text-align: left"><strong>Data Section</strong></td><td style="text-align: center"></td><td style="text-align: center"><code>0x5040_0257</code></td></tr>
<tr><td style="text-align: left"><strong>BSS Section</strong></td><td style="text-align: center"><code>0x5040_0290</code></td><td style="text-align: center"><code>0x5040_6FFF</code></td></tr>
<tr><td style="text-align: left"><strong>Kernel Idle Stack</strong></td><td style="text-align: center"></td><td style="text-align: center"><code>0x5040_7BFF</code></td></tr>
<tr><td style="text-align: left"><strong>Initial RAM Disk</strong></td><td style="text-align: center"><code>0x5040_8288</code></td><td style="text-align: center"><code>0x50BD_8297</code></td></tr>
<tr><td style="text-align: left"><strong>RAM Disk Region</strong></td><td style="text-align: center"><code>0x50A0_0000</code></td><td style="text-align: center"><code>0x519F_FFFF</code></td></tr>
</tbody></table>
</div>
<p>(NuttX will mount the RAM Disk from <strong>RAM Disk Region</strong>)</p>
<p>(Which overlaps with <strong>Initial RAM Disk</strong>!)</p>
<p>This says…</p>
<ol>
<li>
<p>NuttX Kernel <strong><code>nuttx.bin</code></strong> terminates at <strong><code>edata</code></strong>.</p>
<p>(End of Data Section)</p>
</li>
<li>
<p>If we append Initial RAM Disk <strong><code>initrd</code></strong> directly to the end of <strong><code>nuttx.bin</code></strong>…</p>
<p>It will collide with the <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_start.c#L181-L200"><strong>BSS Section</strong></a> and the <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_head.S#L68-L77"><strong>Kernel Idle Stack</strong></a>.</p>
<p>And <strong><code>initrd</code></strong> will get overwritten when NuttX runs the <strong>Boot Code</strong> and <strong>Start Code</strong>.</p>
<p>(Boot Code uses the Kernel Idle Stack. Start Code erases the BSS)</p>
</li>
<li>
<p>Best place to append <strong><code>initrd</code></strong> is after the <strong>Kernel Idle Stack</strong>.</p>
<p>(Roughly <strong>32 KB</strong> after <strong><code>edata</code></strong>)</p>
</li>
<li>
<p>That’s why we inserted a padding of <strong>64 KB</strong> between <strong><code>nuttx.bin</code></strong> and <strong><code>initrd</code></strong>.</p>
<p>(Surely <strong><code>initrd</code></strong> won’t collide with BSS and Kernel Idle Stack)</p>
</li>
<li>
<p>From the previous section, our code locates <strong><code>initrd</code></strong>.</p>
<p>(Searching for the ROM FS Magic Number)</p>
<p>And copies <strong><code>initrd</code></strong> to the <strong>RAM Disk Region</strong>.</p>
</li>
<li>
<p>Finally NuttX mounts the RAM Disk from <strong>RAM Disk Region</strong>.</p>
<p>NuttX Kernel starts the <strong>NuttX Shell</strong> correctly from the Mounted RAM Disk.</p>
<p>(Everything goes well, nothing gets contaminated)</p>
</li>
</ol>
<p>Yep our 64 KB Padding looks legit!</p>
<p><em>64 KB sounds arbitrary. What if the parameters change?</em></p>
<p>We have <strong>Runtime Checks</strong> to catch problems: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_start.c#L136-L170">bl808_start.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Stop if RAM Disk is missing
if (ramdisk_addr == NULL) { _err(&quot;Missing RAM Disk. Check the initrd padding.&quot;); PANIC(); }
// RAM Disk must be after Idle Stack, to prevent overwriting
if (ramdisk_addr &lt;= (uint8_t *)BL808_IDLESTACK_TOP) { _err(&quot;RAM Disk must be after Idle Stack. Increase the initrd padding by %ul bytes.&quot;, (size_t)BL808_IDLESTACK_TOP - (size_t)ramdisk_addr); PANIC(): }
// Filesystem Size must be less than RAM Disk Memory Region
if (size &gt; (size_t)__ramdisk_size) { _err(&quot;RAM Disk Region too small&quot;); PANIC(); }</code></pre></div>
<p><em>Why call bl808_copy_overlap to copy initrd to RAM Disk Region? Why not memcpy?</em></p>
<p>That’s because <strong><code>initrd</code></strong> overlaps with <strong>RAM Disk Region</strong>! (See above)</p>
<p><strong><code>memcpy</code></strong> won’t work with <strong>Overlapping Memory Regions</strong>. Thus we added this: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/bl808/bl808_start.c#L70-L104">bl808_start.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Copy a chunk of memory from `src` to `dest`.
// `dest` overlaps with the end of `src`.
// From libs/libc/string/lib_memmove.c
void *bl808_copy_overlap(void *dest, const void *src, size_t count) {
if (dest &lt;= src) { _err(&quot;dest and src should overlap&quot;); PANIC(); }
char *d = (char *) dest + count;
char *s = (char *) src + count;
// TODO: This needs to be `volatile` or GCC Compiler will replace this by memcpy. Very strange.
while (count--) {
d -= 1; s -= 1;
volatile char c = *s;
*d = c;
}
return dest;
}</code></pre></div>
<p><em>We’re sure that it works?</em></p>
<p>We called <strong><code>verify_image</code></strong> to do a simple Integrity Check on <strong><code>initrd</code></strong>, before and after copying: <a href="https://github.com/lupyuen2/wip-nuttx/blob/ox64a/arch/risc-v/src/jh7110/jh7110_start.c#L236-L248">jh7110_start.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Before Copy: Verify the RAM Disk Image to be copied
verify_image(ramdisk_addr);
// Copy the Filesystem bytes to RAM Disk Memory Region
// Warning: __ramdisk_start overlaps with ramdisk_addr + size
// Which doesn&#39;t work with memcpy.
// Sadly memmove is aliased to memcpy, so we implement memmove ourselves
bl808_copy_overlap((void *)__ramdisk_start, ramdisk_addr, size);
// After Copy: Verify the copied RAM Disk Image
verify_image(__ramdisk_start);</code></pre></div>
<p><a href="https://github.com/lupyuen2/wip-nuttx/blob/ox64a/arch/risc-v/src/jh7110/jh7110_start.c#L248-L455">(<strong><code>verify_image</code></strong> searches for a specific byte)</a></p>
<p>That’s how we discovered that <strong><code>memcpy</code></strong> doesn’t work. And our <strong><code>bl808_copy_overlap</code></strong> works great for the Initial RAM Disk and NuttX Shell! (Pic below)</p>
<p><img src="https://lupyuen.github.io/images/mmu-boot1.png" alt="Ox64 boots to NuttX Shell" /></p>
<p><a href="https://gist.github.com/lupyuen/aa9b3e575ba4e0c233ab02c328221525#file-ox64-nuttx20-log-L115-L323"><em>Ox64 boots to NuttX Shell</em></a></p>
<h1 id="whats-next"><a class="doc-anchor" href="#whats-next">§</a>10 What’s Next</h1>
<p>Like we said at the top of the article…</p>
<blockquote>
<p><em>“One can hide on the First of the Month… But not on the Fifteenth!”</em></p>
</blockquote>
<p>Today we unravelled the inner workings of <strong>NuttX Applications</strong> for <strong>Ox64 BL808 RISC-V SBC</strong>…</p>
<ul>
<li>
<p>We studied the internals of the <strong>simplest NuttX App</strong></p>
</li>
<li>
<p>How NuttX Apps make System Calls with <strong><code>ecall</code></strong>, <strong>Proxy Functions</strong> and <strong>Stub Functions</strong></p>
</li>
<li>
<p>Why NuttX Kernel can access the <strong>Virtual Memory</strong> of NuttX Apps</p>
</li>
<li>
<p>How NuttX Kernel loads <strong>ELF Executables</strong></p>
</li>
<li>
<p>Bundling of NuttX Apps into the <strong>Initial RAM Disk</strong> in ROM FS Format</p>
</li>
<li>
<p>And making sure our RAM Disk is <strong>safe and sound</strong> after loading by U-Boot Bootloader</p>
</li>
</ul>
<p>We’ll do much more for <strong>NuttX on Ox64 BL808</strong>, stay tuned for updates!</p>
<p><a href="https://lupyuen.github.io/articles/ox2#appendix-uart-driver-for-ox64">(Like the fixing of <strong>UART Interrupts</strong>)</a></p>
<p>Many Thanks to my <a href="https://lupyuen.github.io/articles/sponsor"><strong>GitHub Sponsors</strong></a> (and the awesome NuttX Community) for supporting my work! This article wouldn’t have been possible without your support.</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/sponsor"><strong>Sponsor me a coffee</strong></a></p>
</li>
<li>
<p><a href="https://news.ycombinator.com/item?id=38417824"><strong>Discuss this article on Hacker News</strong></a></p>
</li>
<li>
<p><a href="https://forum.pine64.org/showthread.php?tid=18904"><strong>Discuss this article on Pine64 Forum</strong></a></p>
</li>
<li>
<p><a href="https://bbs.bouffalolab.com/d/262-article-risc-v-ox64-bl808-sbc-nuttx-apps-and-initial-ram-disk"><strong>Discuss this article on Bouffalo Lab Forum</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-ox64"><strong>My Current Project: “Apache NuttX RTOS for Ox64 BL808”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-star64"><strong>My Other Project: “NuttX for Star64 JH7110”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx"><strong>Older Project: “NuttX for PinePhone”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io"><strong>Check out my articles</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/rss.xml"><strong>RSS Feed</strong></a></p>
</li>
</ul>
<p><em>Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…</em></p>
<p><a href="https://github.com/lupyuen/lupyuen.github.io/blob/master/src/app.md"><strong>lupyuen.github.io/src/app.md</strong></a></p>
<!-- Begin scripts/rustdoc-after.html: Post-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker and Prism Theme -->
<script src="../theme.js"></script>
<script src="../prism.js"></script>
<!-- Theme Picker and Prism Theme -->
<!-- End scripts/rustdoc-after.html -->
</body>
</html>
Reference in a new issue View git blame Copy permalink