lupyuen/lupyuen.org
1
0
Fork 0
mirror of https://github.com/lupyuen/lupyuen.github.io.git synced 2025-01-13 10:18:33 +08:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
lupyuen.org/articles/bisect.html
Lup Yuen Lee d4d082511d
Some checks are pending
Build Articles / build (push) Waiting to run
Details
Commit from GitHub Actions
2025-01-07 10:37:33 +00:00

713 lines
No EOL
44 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="rustdoc">
<title>Git Bisecting a Bug (Apache NuttX RTOS)</title>
<!-- Begin scripts/articles/*-header.html: Article Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<meta property="og:title"
content="Git Bisecting a Bug (Apache NuttX RTOS)"
data-rh="true">
<meta property="og:description"
content="Suppose we hit a Runtime Bug in Apache NuttX RTOS. We think that the Breaking Commit (causing the bug) is somewhere inside these hundreds of NuttX Commits. But which one? In this article: We run Git Bisect to discover the Breaking Commit."
data-rh="true">
<meta name="description"
content="Suppose we hit a Runtime Bug in Apache NuttX RTOS. We think that the Breaking Commit (causing the bug) is somewhere inside these hundreds of NuttX Commits. But which one? In this article: We run Git Bisect to discover the Breaking Commit.">
<meta property="og:image"
content="https://lupyuen.github.io/images/bisect-title.jpg">
<meta property="og:type"
content="article" data-rh="true">
<link rel="canonical"
href="https://lupyuen.org/articles/bisect.html" />
<!-- End scripts/articles/*-header.html -->
<!-- Begin scripts/rustdoc-header.html: Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<link rel="alternate" type="application/rss+xml" title="RSS Feed for lupyuen" href="/rss.xml" />
<link rel="stylesheet" type="text/css" href="../normalize.css">
<link rel="stylesheet" type="text/css" href="../rustdoc.css" id="mainThemeStyle">
<link rel="stylesheet" type="text/css" href="../dark.css">
<link rel="stylesheet" type="text/css" href="../light.css" id="themeStyle">
<link rel="stylesheet" type="text/css" href="../prism.css">
<script src="../storage.js"></script><noscript>
<link rel="stylesheet" href="../noscript.css"></noscript>
<link rel="shortcut icon" href="../favicon.ico">
<style type="text/css">
#crate-search {
background-image: url("../down-arrow.svg");
}
</style>
<!-- End scripts/rustdoc-header.html -->
</head>
<body class="rustdoc">
<!--[if lte IE 8]>
<div class="warning">
This old browser is unsupported and will most likely display funky
things.
</div>
<![endif]-->
<!-- Begin scripts/rustdoc-before.html: Pre-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker -->
<div class="theme-picker" style="left: 0"><button id="theme-picker" aria-label="Pick another theme!"><img src="../brush.svg"
width="18" alt="Pick another theme!"></button>
<div id="theme-choices"></div>
</div>
<!-- Theme Picker -->
<!-- End scripts/rustdoc-before.html -->
<h1 class="title">Git Bisecting a Bug (Apache NuttX RTOS)</h1>
<nav id="rustdoc"><ul>
<li><a href="#automated-bisect" title="Automated Bisect">1 Automated Bisect</a><ul></ul></li>
<li><a href="#simulate-the-git-bisect" title="Simulate The Git Bisect">2 Simulate The Git Bisect</a><ul></ul></li>
<li><a href="#continuous-integration-test" title="Continuous Integration Test">3 Continuous Integration Test</a><ul></ul></li>
<li><a href="#git-bisect-for-real" title="Git Bisect For Real">4 Git Bisect For Real</a><ul></ul></li>
<li><a href="#git-bisect-gets-quirky" title="Git Bisect Gets Quirky">5 Git Bisect Gets Quirky</a><ul></ul></li>
<li><a href="#good-commit-goes-bad" title="Good Commit Goes Bad">6 Good Commit Goes Bad</a><ul></ul></li>
<li><a href="#fixing-the-bug" title="Fixing The Bug">7 Fixing The Bug</a><ul></ul></li>
<li><a href="#increase-the-stack" title="Increase The Stack">8 Increase The Stack</a><ul></ul></li>
<li><a href="#whats-next" title="What’s Next">9 What’s Next</a><ul></ul></li>
<li><a href="#appendix-inspect-executable-size-with-bloaty" title="Appendix: Inspect Executable Size with Bloaty">10 Appendix: Inspect Executable Size with Bloaty</a><ul></ul></li></ul></nav><p>📝 <em>5 Jan 2025</em></p>
<p><img src="https://lupyuen.github.io/images/bisect-title.jpg" alt="Git Bisecting a Bug in Apache NuttX RTOS" /></p>
<span style="font-size:80%">
<blockquote>
<p><em>“Because I’m bad, I’m bad, shamone <br> (bad, bad, really, really bad) <br>
You know I’m bad, I’m bad (bad, bad) … <br>
And the whole world has to answer right now <br>
Just to tell you once again <br>
<a href="https://en.wikipedia.org/wiki/Bad_(Michael_Jackson_song)">Who’s bad?</a>”</em></p>
</blockquote>
</span>
<p>2 Weeks Ago: We saw a <a href="https://lupyuen.github.io/articles/ci7"><strong>Runtime Bug</strong></a> in <strong>Apache NuttX RTOS</strong>. We think that the <strong>Breaking Commit</strong> (causing the bug) falls somewhere between these <a href="https://docs.google.com/spreadsheets/d/1aNu1OensFc-QA1EfyTe6CcbfduzR3gdbbnZfRTca0fw/edit?gid=0#gid=0"><strong>“Good” and “Bad” Commits</strong></a>…</p>
<div><table><thead><tr><th style="text-align: center"></th><th style="text-align: center"></th></tr></thead><tbody>
<tr><td style="text-align: center"><a href="https://github.com/apache/nuttx/issues/14808#issue-2661180633"><strong>Commit #1 is Good</strong></a> <br> <em>2024-11-14</em></td><td style="text-align: center">NuttX runs OK <br> <a href="https://github.com/apache/nuttx/tree/6554ed4d668e0c3982aaed8d8fb4b8ae81e5596c"><em>6554ed4</em></a></td></tr>
<tr><td style="text-align: center"><em>[ many commits ]</em></td><td style="text-align: center"><em>…</em></td></tr>
<tr><td style="text-align: center"><a href="https://github.com/apache/nuttx/issues/14808#issuecomment-2518119367"><strong>Commit #468 is Bad</strong></a> <br> <em>2024-12-04</em></td><td style="text-align: center">NuttX hits Error <br> <a href="https://github.com/apache/nuttx/tree/79a1ebb9cd0c13f48a57413fa4bc3950b2cd5e0b"><em>79a1ebb</em></a></td></tr>
</tbody></table>
</div>
<p>That’s <a href="https://docs.google.com/spreadsheets/d/1aNu1OensFc-QA1EfyTe6CcbfduzR3gdbbnZfRTca0fw/edit?gid=0#gid=0"><strong>468 Commits</strong></a>. Which one is the Breaking Commit?</p>
<p><em>Maybe we Rewind Each Commit and test?</em></p>
<p>With a script, we could rewind and retest 468 Commits for <a href="https://lupyuen.github.io/articles/ci6"><strong>Compile Errors</strong></a>. But it’s probably too slow for <strong>Runtime Errors</strong>. <em>(Rewind + Recompile + Rerun)</em></p>
<p>We have a quicker way: <strong>Git Bisect</strong>! (Pic above)</p>
<h1 id="automated-bisect"><a class="doc-anchor" href="#automated-bisect">§</a>1 Automated Bisect</h1>
<p><em>What’s this Git Bisect?</em></p>
<p>Remember <a href="https://en.wikipedia.org/wiki/Binary_search"><strong>Binary Chop</strong></a>?</p>
<blockquote>
<p><em>“I’m thinking of A Number <br> Guess My Number! <br> It’s from 1 to 468 <br> Ask me 9 Yes-No Questions”</em></p>
</blockquote>
<p>To solve this, we <strong>Divide And Conquer</strong>: Is 234 too high? <em>(no)</em> Is 351 too high? <em>(yes)</em> Is 292 too high <em>(yes)</em>…</p>
<p><img src="https://lupyuen.github.io/images/bisect-guess.jpg" alt="Binary Chop guessing my number" /></p>
<p><a href="https://git-scm.com/docs/git-bisect"><strong>Git Bisect</strong></a> works the same way, but for <strong>Git Commits</strong>…</p>
<ul>
<li>
<p>Our <strong>Breaking Commit</strong> is one of 468 Commits</p>
</li>
<li>
<p>Git Bisect shall <strong>Pick the Middle Commit</strong> and ask: “Is this a Good Commit or Bad Commit?”</p>
</li>
<li>
<p>Repeat until it discovers the <strong>Breaking Commit</strong> (in 9 steps)</p>
</li>
</ul>
<p><em>Is it automated?</em></p>
<p>Yep Git Bisect will gleefully seek the Breaking Commit on its own… Assuming that we provide a Script to <strong>Assess the Goodness / Badness</strong> of a NuttX Commit: <a href="https://github.com/lupyuen/nuttx-bisect/blob/main/my-test-script.sh">my-test-script.sh</a></p>
<div class="example-wrap"><pre class="language-bash"><code>## This script will be called by Git Bisect
## In Case of Error: Return the error to Git Bisect
set -e
## Get the NuttX Hash (Commit ID)
nuttx_hash=$(git rev-parse HEAD)
## For the NuttX Commit:
## We Build, Run and Test the NuttX Commit...
## make distclean || true
## tools/configure.sh ox64:nsh
## make -j
## But for now: We randomly simulate OK or Error
random_0_or_1=$(( $RANDOM % 2 ))
if [[ &quot;$random_0_or_1&quot; == &quot;0&quot; ]]; then
exit 0 ## Simulate OK
else
exit 1 ## Simulate Error
fi
## Beware: Don&#39;t return Exit Codes above 124!
## https://git-scm.com/docs/git-bisect#_bisect_run</code></pre></div>
<p><a href="https://git-scm.com/docs/git-bisect#_basic_bisect_commands_start_bad_good">(Or we <strong>Bisect Manually</strong>)</a></p>
<p>This is how we start our <strong>Simulated Git Bisect</strong>: <a href="https://github.com/lupyuen/nuttx-bisect/blob/main/run.sh">run.sh</a></p>
<div class="example-wrap"><pre class="language-bash"><code>## Download the NuttX Repo and NuttX Apps
git clone https://github.com/apache/nuttx
git clone https://github.com/apache/nuttx-apps apps
cd nuttx
## Tell Git Bisect the Good and Bad Commits
## (Or specify HEAD for the Latest Commit)
git bisect start
git bisect good 6554ed4 ## Commit #1 is Good
git bisect bad 79a1ebb ## Commit #468 is Bad
## Bisect with our Simulated Test Script
git bisect run \
$HOME/nuttx-bisect/my-test-script.sh
## Commit #235 is the Breaking Commit:
## 74bac56 is the first bad commit</code></pre></div>
<p><a href="https://gist.github.com/lupyuen/160613f2b68f1ab81f1c46146c189b9f">(See the <strong>Complete Log</strong>)</a></p>
<p>That was quick! We break it down…</p>
<p><img src="https://lupyuen.github.io/images/bisect-screen1.png" alt="Simulating The Git Bisect" /></p>
<h1 id="simulate-the-git-bisect"><a class="doc-anchor" href="#simulate-the-git-bisect">§</a>2 Simulate The Git Bisect</h1>
<p><em>What just happened in our Simulated Git Bisect?</em></p>
<ol>
<li>
<p>Git Bisect picked the <a href="https://gist.github.com/lupyuen/160613f2b68f1ab81f1c46146c189b9f#file-gistfile1-txt-L38-L69"><strong>Middle Commit #<code>234</code></strong></a>…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Testing Commit #234 (94a2ce3)
$HOME/nuttx-bisect/my-test-script.sh
nuttx_hash=94a2ce3
## Our Script simulates a successful test
exit 0 ## Simulate OK</code></pre></div>
<p>And discovered that <strong>Commit #<code>234</code> is Good</strong>. (Via our Simulated Script)</p>
</li>
<li>
<p>Then it continued the <strong>Simulated Bisecting</strong> (coincidentally, all bad)</p>
<div class="example-wrap"><pre class="language-bash"><code>## Commit #351 is Bad
nuttx_hash=1cfaff0
exit 1 ## Simulate Error
## Commit #292 is Bad
nuttx_hash=65a93e9
exit 1 ## Simulate Error
## Commit #263 is Bad
nuttx_hash=1e265af
exit 1 ## Simulate Error
## Commit #248 is Bad
nuttx_hash=c70f3e3
exit 1 ## Simulate Error
## Commit #241 is Bad
nuttx_hash=5d86bee
exit 1 ## Simulate Error
## Commit #237 is Bad
nuttx_hash=e7c2e7c
exit 1 ## Simulate Error
## Commit #236 is Bad
nuttx_hash=68d47ee
exit 1 ## Simulate Error
## Commit #235 is Bad
nuttx_hash=74bac56
exit 1 ## Simulate Error</code></pre></div>
<p><a href="https://gist.github.com/lupyuen/160613f2b68f1ab81f1c46146c189b9f">(See the <strong>Complete Log</strong>)</a></p>
</li>
<li>
<p>Finally deducing that <strong>Commit #<code>235</code></strong> is the <strong>Breaking Commit</strong></p>
<div class="example-wrap"><pre class="language-bash"><code>## Commit #235 is the Breaking Commit
74bac56 is the first bad commit</code></pre></div></li>
</ol>
<p>This works fine for our <strong>Simulated Git Bisect</strong>. Now we do it for real…</p>
<p><a href="https://github.com/lupyuen/nuttx-bisect/blob/main/my-test-script.sh#L34-L76">(OK maybe it’s <strong>no coincidence</strong>)</a></p>
<p><img src="https://lupyuen.github.io/images/bisect-screen2.png" alt="Docker running CI Test in CI Job risc-v-05" /></p>
<h1 id="continuous-integration-test"><a class="doc-anchor" href="#continuous-integration-test">§</a>3 Continuous Integration Test</h1>
<p><em>Will Git Bisect work for Real-Life NuttX?</em></p>
<p>From our <a href="https://github.com/apache/nuttx/issues/14808"><strong>Bug Report</strong></a>: NuttX fails the <strong>Continuous Integration Test</strong> (CI Test) for RISC-V QEMU…</p>
<div class="example-wrap"><pre class="language-text"><code>Configuration/Tool: rv-virt/citest
test_cmocka PASSED
test_hello PASSED
test_helloxx FAILED
test_pipe FAILED
test_usrsocktest FAILED
[...Failing all the way...]</code></pre></div>
<p>This happens inside the <strong>CI Job <em>risc-v-05</em></strong>. Which we can reproduce with <strong>Docker Engine</strong>: <a href="https://github.com/lupyuen/nuttx-bisect/blob/main/run-job-bisect.sh#L36-L61">run-job-bisect.sh</a></p>
<div class="example-wrap"><pre class="language-bash"><code>## TODO: Install Docker Engine
## https://docs.docker.com/engine/install/ubuntu/
## Assume we&#39;re running risc-v-05 with
## Latest NuttX Repo and NuttX Apps
job=risc-v-05
nuttx_hash=HEAD
apps_hash=HEAD
## Run the CI Job in Docker Container
## If CI Test Hangs: Kill it after 1 hour
sudo docker run -it \
ghcr.io/apache/nuttx/apache-nuttx-ci-linux:latest \
/bin/bash -c &quot;
set -e ;
set -x ;
uname -a ;
cd ;
pwd ;
git clone https://github.com/apache/nuttx ;
git clone https://github.com/apache/nuttx-apps apps ;
echo Building nuttx @ $nuttx_hash / nuttx-apps @ $apps_hash ;
pushd nuttx ; git reset --hard $nuttx_hash ; popd ;
pushd apps ; git reset --hard $apps_hash ; popd ;
pushd nuttx ; echo NuttX Source: https://github.com/apache/nuttx/tree/\$(git rev-parse HEAD) ; popd ;
pushd apps ; echo NuttX Apps: https://github.com/apache/nuttx-apps/tree/\$(git rev-parse HEAD) ; popd ;
sleep 10 ;
cd nuttx/tools/ci ;
( sleep 3600 ; echo Killing pytest after timeout... ; pkill -f pytest )&amp;
(
(./cibuild.sh -c -A -N -R testlist/$job.dat) || (res=\$? ; echo &#39;***** JOB FAILED&#39; ; exit \$res)
)
&quot;</code></pre></div>
<p><a href="https://lupyuen.codeberg.page/articles/ci2.html#build-nuttx-for-one-target-group">(More about <strong>Docker Builds for NuttX</strong>)</a></p>
<p>Everything above becomes our <strong>Git Bisect Script</strong> that assesses “Goodness” vs “Badness” for a NuttX Commit: <a href="https://github.com/lupyuen/nuttx-bisect/blob/main/run-job-bisect.sh#L36-L73">run-job-bisect.sh</a></p>
<div class="example-wrap"><pre class="language-bash"><code>## This script will be called by Git Bisect Wrapper...
## We run the CI Job in Docker Container
## (Copy from above)
sudo docker run -it ...
## Result the result to the caller
res=$?
if [[ &quot;$res&quot; == &quot;0&quot; ]]; then
exit 0 ## Return OK
else
exit 1 ## Return Error
fi
## Beware: Don&#39;t return Exit Codes above 124!
## https://git-scm.com/docs/git-bisect#_bisect_run</code></pre></div>
<p>Which is called by our <strong>Git Bisect Wrapper</strong>: <a href="https://github.com/lupyuen/nuttx-bisect/blob/main/start-job-bisect.sh">start-job-bisect.sh</a></p>
<div class="example-wrap"><pre class="language-bash"><code>## This wrapper script will be called by Git Bisect
## Must be run as `sudo`! (Because of Docker)
## Get the NuttX Hash (Commit ID)
nuttx_hash=$(git rev-parse HEAD)
## Run the CI Job for the NuttX Commit
## Passing the Job Name, NuttX Hash and Apps Hash
## (Or set Apps Hash to HEAD for the Latest Commit)
job=risc-v-05
apps_hash=1c7a7f7
$HOME/nuttx-bisect/run-job-bisect.sh \
$job $nuttx_hash $apps_hash
## This Git Bisect script will work for any CI Job!
## Just change `job=risc-v-05` to the CI Job Name (like arm-01)</code></pre></div>
<p>We’re ready to run this!</p>
<p><img src="https://lupyuen.github.io/images/bisect-screen3.png" alt="Running Git Bisect on Real NuttX Commits" /></p>
<h1 id="git-bisect-for-real"><a class="doc-anchor" href="#git-bisect-for-real">§</a>4 Git Bisect For Real</h1>
<p><em>What happens in Git Bisect?</em></p>
<ol>
<li>
<p>We start Git Bisect, telling it that Commit #<code>1</code> is Good and Commit #<code>468</code> is Bad: <a href="https://github.com/lupyuen/nuttx-bisect/blob/main/run2.sh">run2.sh</a></p>
<div class="example-wrap"><pre class="language-bash"><code>sudo --shell ## Needed by Docker
git clone https://github.com/apache/nuttx
cd nuttx
git bisect start
git bisect good 6554ed4 ## Commit #1
git bisect bad 79a1ebb ## Commit #468
git bisect run \
$HOME/nuttx-bisect/start-job-bisect.sh</code></pre></div></li>
<li>
<p>Git Bisect picks the <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L42-L74"><strong>Middle Commit #<code>234</code></strong></a> and runs our script</p>
<div class="example-wrap"><pre class="language-bash"><code>## Testing Commit #234 (94a2ce3)
## For CI Job risc-v-05 (CI Test for RISC-V QEMU)
## With NuttX Apps (1c7a7f7)
$HOME/nuttx-bisect/run-job-bisect.sh \
risc-v-05 \
94a2ce3 \
1c7a7f7</code></pre></div></li>
<li>
<p>And discovers that <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L1260-L1379"><strong>Commit #<code>234</code> is Good</strong></a> (via our script)</p>
<div class="example-wrap"><pre class="language-bash"><code>## Commit #234: Completed CI Test successfully
Configuration/Tool: rv-virt/citest
test_ostest PASSED
exit 0</code></pre></div></li>
<li>
<p>Beware: Every Bisect will be <strong>Super Slow</strong>! Whenever something fails: CI Test will <a href="https://lupyuen.github.io/articles/ci7#dump-the-ci-log-file"><strong>hang the CI Job</strong></a>.</p>
<p>(Our script will kill CI Test after 1 hour)</p>
</li>
<li>
<p>Then it continues bisecting. Assessing Commits <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L1386-L1420">#<code>351</code></a>, <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L1876-L1912">#<code>292</code></a>, <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L2367-L2405">#<code>263</code></a>, <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L3028-L3068">#<code>248</code></a>, <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L3524-L3566">#<code>241</code></a>, <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L3935-L3979">#<code>237</code></a>, <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L4325-L4371">#<code>236</code></a>, <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L4683-L4731">#<code>235</code></a></p>
<div class="example-wrap"><pre class="language-bash"><code>## Commit #351 is Bad
run-job-bisect.sh ... 1cfaff0 ...
test_ltp_interfaces_pthread_barrierattr_init_2_1 FAILED
exit 1
## Commit #292 is Bad
run-job-bisect.sh ... 65a93e9 ...
test_ltp_interfaces_pthread_barrierattr_init_2_1 FAILED
exit 1
## Commit #263 is Bad
run-job-bisect.sh ... 1e265af ...
test_ltp_interfaces_sigrelse_1_1 FAILED
exit 1
## Commit #248 is Bad
run-job-bisect.sh ... c70f3e3 ...
test_ltp_interfaces_pthread_barrierattr_init_2_1 FAILED
exit 1
## Commit #241 is Bad
run-job-bisect.sh ... 5d86bee ...
test_ltp_interfaces_mq_open_7_3 FAILED
exit 1
## Commit #237 is Bad
run-job-bisect.sh ... e7c2e7c ...
test_ltp_interfaces_sigaction_23_7 FAILED
exit 1
## Commit #236 is Bad
run-job-bisect.sh ... 68d47ee ...
test_ltp_interfaces_pthread_getcpuclockid_1_1 FAILED
exit 1
## Commit #235 is Bad
run-job-bisect.sh ... 74bac56 ...
test_ltp_interfaces_pthread_detach_1_1 FAILED
exit 1</code></pre></div>
<p><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d">(See the <strong>Complete Log</strong>)</a></p>
</li>
<li>
<p>Which says in 9 steps (pic below)</p>
<div><table><thead><tr><th style="text-align: center"></th><th style="text-align: center"></th></tr></thead><tbody>
<tr><td style="text-align: center"><em>Commit #468</em></td><td style="text-align: center"><em>Is Bad</em></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L42-L74"><strong>Commit #<code>234</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L1260-L1379"><strong>Is Good</strong></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L1386-L1420"><strong>Commit #<code>351</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L1789-L1869"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L1876-L1912"><strong>Commit #<code>292</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L2281-L2360"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L2367-L2405"><strong>Commit #<code>263</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L2949-L3021"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L3028-L3068"><strong>Commit #<code>248</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L3437-L3517"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L3524-L3566"><strong>Commit #<code>241</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L3845-L3928"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L3935-L3979"><strong>Commit #<code>237</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L4235-L4318"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L4325-L4371"><strong>Commit #<code>236</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L4591-L4676"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L4683-L4731"><strong>Commit #<code>235</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L5087-L5166"><em>Is Bad</em></a> <br> <em>(really really)</em></td></tr>
</tbody></table>
</div></li>
<li>
<p>Finally deducing that <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L4683-L4731"><strong>Commit #<code>235</code></strong></a> is the <a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L5167-L5210"><strong>Breaking Commit</strong></a> (pic below)</p>
<div class="example-wrap"><pre class="language-bash"><code>## Commit #235 is the Breaking Commit
74bac56 is the first bad commit</code></pre></div>
<p><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d">(See the <strong>Complete Log</strong>)</a></p>
</li>
</ol>
<p><img src="https://lupyuen.github.io/images/bisect-title.jpg" alt="Git Bisecting a Bug in Apache NuttX RTOS" /></p>
<p><em>How long did Git Bisect run?</em></p>
<p><a href="https://gist.github.com/lupyuen/39cdb916d30625388974e00d5daa676d#file-gistfile1-txt-L216-L4873"><strong>8 Hours!</strong></a> I left it simmering overnight. (Just like my Six-Bean Stew)</p>
<p>Remember that Every Bisect takes <strong>1 Hour to Recompile + Rerun</strong>, because of the Stuck CI Test. That’s why Git Bisect works especially well for slower jobs.</p>
<h1 id="git-bisect-gets-quirky"><a class="doc-anchor" href="#git-bisect-gets-quirky">§</a>5 Git Bisect Gets Quirky</h1>
<p><em>Did Git Bisect find the correct Breaking Commit?</em></p>
<p>To be <em>really really</em> sure: We run Git Bisect <a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493"><strong>one more time</strong></a>…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Restart the Git Bisect (`sudo` is needed by Docker)
sudo --shell
git bisect start
git bisect good 6554ed4 ## Commit #1
git bisect bad 79a1ebb ## Commit #468
git bisect run \
$HOME/nuttx-bisect/start-job-bisect.sh</code></pre></div>
<p>Here comes the twist, watch carefully…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Commit #234 is Bad
run-job-bisect.sh ... 94a2ce3 ...
test_ltp_interfaces_mq_close_3_2 FAILED
exit 1
## Commit #117 is Good
run-job-bisect.sh ... 96a3bc2 ...
test_ostest PASSED
exit 0
## Commit #138 is Bad
run-job-bisect.sh ... 3a46b6e ...
test_ltp_interfaces_pthread_barrierattr_init_2_1 FAILED
exit 1
## Commit #146 is Bad
run-job-bisect.sh ... dac3f31 ...
test_ltp_interfaces_sigaction_6_3 FAILED
exit 1
## Commit #131 is Good
run-job-bisect.sh ... 4c3ae2e ...
test_ostest PASSED
exit 0
## Commit #138 is Bad
run-job-bisect.sh ... 3b50bf1 ...
test_ltp_interfaces_pthread_barrierattr_init_2_1 FAILED
exit 1
## Commit #134 is Bad
run-job-bisect.sh ... 5ff98f6 ...
test_ltp_interfaces_pthread_barrierattr_init_2_1 FAILED
exit 1
## Commit #133 is Bad
run-job-bisect.sh ... b4d8ac8 ...
test_ltp_interfaces_pthread_barrierattr_init_2_1 FAILED
exit 1
## Commit #132 is Bad
run-job-bisect.sh ... fb92b60 ...
test_ltp_interfaces_pthread_barrierattr_init_2_1 FAILED
exit 1</code></pre></div>
<p><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493">(See the <strong>Complete Log</strong>)</a></p>
<p>Which deduces…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Commit #132 is the Breaking Commit
fb92b60 is the first bad commit
## Previously: Commit #235 is the Breaking Commit!
## 74bac56 is the first bad commit</code></pre></div>
<p><strong>Commit #<code>132</code></strong> is now the Breaking Commit, not Commit #<code>235</code>!</p>
<p>Hmmm something has changed. Why?</p>
<div><table><thead><tr><th style="text-align: center"></th><th style="text-align: center"></th></tr></thead><tbody>
<tr><td style="text-align: center"><em>Commit #468</em></td><td style="text-align: center"><em>Is Bad</em></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L46-L78"><strong>Commit #<code>234</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L337-L421"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L428-L462"><strong>Commit #<code>117</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L1648-L1767"><strong>Is Good</strong></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L1774-L1810"><strong>Commit #<code>138</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L2179-L2259"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L2266-L2304"><strong>Commit #<code>146</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L2646-L2726"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L2733-L2773"><strong>Commit #<code>131</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L3959-L4078"><strong>Is Good</strong></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L4085-L4127"><strong>Commit #<code>138</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L4496-L4575"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L4582-L4626"><strong>Commit #<code>134</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L4995-L5074"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L5081-L5127"><strong>Commit #<code>133</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L5496-L5575"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L5582-L5630"><strong>Commit #<code>132</code></strong></a></td><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L5999-L6078"><em>Is Bad</em></a></td></tr>
<tr><td style="text-align: center"><a href="https://gist.github.com/lupyuen/5a92fb9ea76751a54d2a82ba0341c493#file-gistfile1-txt-L6079-L6123"><strong>Commit #<code>132</code></strong></a></td><td style="text-align: center">Is the <strong>Breaking Commit</strong> <br> <em>(really really?)</em></td></tr>
</tbody></table>
</div>
<p><img src="https://lupyuen.github.io/images/bisect-screen4.png" alt="Good Commit #234 Goes Bad" /></p>
<h1 id="good-commit-goes-bad"><a class="doc-anchor" href="#good-commit-goes-bad">§</a>6 Good Commit Goes Bad</h1>
<p><em>Why is Git Bisect telling us a different Breaking Commit?</em></p>
<p>In The Movies: Arnold travels to the past <em>(in a Time Machine)</em>, changing something in history, and the future changes.</p>
<p>In Real Life: <strong>Commit #<code>234</code></strong> has changed in history. Altering our future! (Pic above)</p>
<div class="example-wrap"><pre class="language-bash"><code>## Previously...
## Commit #234 is Good
run-job-bisect.sh ... 94a2ce3 ...
test_ostest PASSED
exit 0
## But Now...
## Commit #234 is Bad
run-job-bisect.sh ... 94a2ce3 ...
test_ltp_interfaces_mq_close_3_2 FAILED
exit 1</code></pre></div>
<p>After this, everything changed. Concluding with a <strong>Different Breaking Commit</strong>. (Think “alternate timeline”)</p>
<p><em>Huh! How did Commit #234 change?</em></p>
<p>This CI Test looks more complicated than we thought. CI Test appears to be <strong>failing with the slightest change</strong> in QEMU Memory. For a Specific Commit: Our bug isn’t reliably reproducible.</p>
<p><strong>Lesson Learnt:</strong> Git Bisect works best for bugs that are <strong>reliably reproducible</strong> for a specified commit!</p>
<p><em>Can we use Git Bisect with Real Hardware? On an Actual NuttX Device?</em></p>
<p>Yep sure Git Bisect will work with any NuttX Device that can be <strong>controlled by a script</strong>.</p>
<p>For Example: <strong>SG2000 RISC-V SBC</strong> has a script for Building NuttX and Booting via TFTP (which can be controlled by Git Bisect)</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/sg2000a"><strong>“Daily Automated Testing for Milk-V Duo S RISC-V SBC”</strong></a></li>
</ul>
<p>Though Honestly: <strong>SG2000 Emulator</strong> would be much quicker (and more reliable) for Git Bisect…</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/sg2000b"><strong>“RISC-V Emulator for Sophgo SG2000 SoC (Pine64 Oz64 / Milk-V Duo S)”</strong></a></p>
<p>(No script? Try <a href="https://git-scm.com/docs/git-bisect#_basic_bisect_commands_start_bad_good"><strong>Manual Git Bisect</strong></a>)</p>
</li>
</ul>
<p><img src="https://lupyuen.github.io/images/bisect-issues.jpg" alt="We have Two Bugs stacked together" /></p>
<h1 id="fixing-the-bug"><a class="doc-anchor" href="#fixing-the-bug">§</a>7 Fixing The Bug</h1>
<p><em>OK so Git Bisect wasn’t 100% successful. How did we fix the bug?</em></p>
<p>Actually we have <strong>Two Bugs</strong> stacked together…</p>
<ul>
<li>
<p><a href="https://github.com/apache/nuttx/issues/14808"><strong>“rv-virt/citest: test_hello or test_pipe failed”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/apache/nuttx/issues/15170"><strong>“rv-virt/citest fails at ltp_interfaces_pthread_barrierattr_init_2_1”</strong></a></p>
</li>
</ul>
<p>The First Bug was a Stack Overflow that <strong>Tiago Medicci Serrano</strong> kindly fixed by increasing the <strong>Init Task Stack Size</strong>…</p>
<ul>
<li><a href="https://github.com/apache/nuttx/pull/15165"><strong>“rv-virt/citest: Increase init task stack size to 3072”</strong></a></li>
</ul>
<p>The Second Bug? Not so obvious which Stack Overflowed…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Start the NuttX Docker Image
sudo docker run \
-it \
ghcr.io/apache/nuttx/apache-nuttx-ci-linux:latest \
/bin/bash
## Run the CI Test in Docker
cd
git clone https://github.com/apache/nuttx
git clone https://github.com/apache/nuttx-apps apps
pushd nuttx ; echo NuttX Source: https://github.com/apache/nuttx/tree/$(git rev-parse HEAD) ; popd
pushd apps ; echo NuttX Apps: https://github.com/apache/nuttx-apps/tree/$(git rev-parse HEAD) ; popd
cd nuttx/tools/ci
./cibuild.sh -c -A -N -R testlist/risc-v-05.dat
## Wait for it to fail
## Press Ctrl-C a few times to stop it
## Then dump the log
cat /root/nuttx/boards/risc-v/qemu-rv/rv-virt/configs/citest/logs/rv-virt/qemu/*
## Output shows: Stack Overflow for ltp_interfaces_pthread_barrierattr_init_2_1
nsh&gt; ltp_interfaces_pthread_barrierattr_init_2_1
riscv_exception: EXCEPTION: Load access fault. MCAUSE: 00000005, EPC: 800074c6, MTVAL: 000002a4
STACKSIZE USED FILLED COMMAND
3936 3936 100.0%! ltp_interfaces_pthread_barriera</code></pre></div>
<p><a href="https://gist.github.com/lupyuen/4ec372cea171b99ae5bc5603aa75a6a7">(See the <strong>Complete Log</strong>)</a></p>
<p><img src="https://lupyuen.github.io/images/bisect-disassembly.png" alt="Searching the NuttX Disassembly for ltp_interfaces_pthread_barrierattr_init_2_1" /></p>
<h1 id="increase-the-stack"><a class="doc-anchor" href="#increase-the-stack">§</a>8 Increase The Stack</h1>
<p><em>What’s ltp_interfaces_pthread_barrierattr_init_2_1? Why is the Stack Overflowing?</em></p>
<p>We search the <strong>NuttX Disassembly</strong> (pic above)</p>
<div class="example-wrap"><pre class="language-bash"><code>## Dump the disassembly to nuttx.S
cd /root/nuttx
riscv-none-elf-objdump \
--syms --source --reloc --demangle --line-numbers --wide \
--debugging \
nuttx \
&gt;nuttx.S \
2&gt;&amp;1
## Search the disassembly for ltp_interfaces_pthread_barrierattr_init_2_1
grep nuttx.S \
ltp_interfaces_pthread_barrierattr_init_2_1</code></pre></div>
<p><a href="https://github.com/lupyuen/nuttx-bisect/releases/download/main-1/nuttx.S">(See the <strong>NuttX Disassembly</strong>)</a></p>
<p>And we see…</p>
<div class="example-wrap"><pre class="language-text"><code>8006642c &lt;ltp_interfaces_pthread_barrierattr_init_2_1_main&gt;:
ltp_interfaces_pthread_barrierattr_init_2_1_main():
apps/testing/ltp/ltp/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_init</code></pre></div>
<p>Aha! It points to a <strong>NuttX Test App</strong>: <a href="https://github.com/apache/nuttx-apps/tree/master/testing/ltp">nuttx-apps/testing/ltp</a></p>
<p>Thus we edit the <strong>Stack Configuration</strong>: <a href="https://github.com/apache/nuttx-apps/tree/master/testing/ltp/Kconfig">testing/ltp/Kconfig</a></p>
<div class="example-wrap"><pre class="language-yaml"><code>## Before: Stack Size is 4 KB
config TESTING_LTP_STACKSIZE
int &quot;Linux Test Project stack size&quot;
default 4096</code></pre></div>
<p>And we double the <strong>Stack Size to 8 KB</strong>…</p>
<div class="example-wrap"><pre class="language-yml"><code> ## After: Stack Size is 8 KB
default 8192</code></pre></div>
<p>We retest in Docker. And our <a href="https://gist.github.com/lupyuen/3688826ed676971536249509ceefe834"><strong>CI Test succeeds</strong></a> yay!</p>
<ul>
<li><a href="https://github.com/apache/nuttx-apps/pull/2888"><strong>“testing/ltp: Increase Stack Size”</strong></a></li>
</ul>
<p><em>But why did we run out of Stack Space? Has something grown too big?</em></p>
<p>We could run <strong>Bloaty</strong> to do detailed analysis of the <strong>Code and Data Size</strong>…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/bisect#appendix-inspect-executable-size-with-bloaty"><strong>“Inspect Executable Size with Bloaty”</strong></a></li>
</ul>
<p><img src="https://lupyuen.github.io/images/bisect-title.jpg" alt="Git Bisecting a Bug in Apache NuttX RTOS" /></p>
<h1 id="whats-next"><a class="doc-anchor" href="#whats-next">§</a>9 What’s Next</h1>
<p>Next Article: What would NuttX Life be like without GitHub? We try out (self-hosted open-source) <strong>Forgejo Git Forge</strong> with NuttX.</p>
<p>After That: Why <strong>Sync-Build-Ingest</strong> is super important for NuttX CI. And how we monitor it with our <strong>Magic Disco Light</strong>.</p>
<p>After After That: Since we can <strong>Rewind NuttX Builds</strong> and automatically <strong>Git Bisect</strong>… Can we create a Bot that will fetch the <strong>Failed Builds from NuttX Dashboard</strong>, identify the Breaking PR, and escalate to the right folks?</p>
<p>Many Thanks to the awesome <strong>NuttX Admins</strong> and <strong>NuttX Devs</strong>! And <a href="https://lupyuen.github.io/articles/sponsor"><strong>My Sponsors</strong></a>, for sticking with me all these years.</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/sponsor"><strong>Sponsor me a coffee</strong></a></p>
</li>
<li>
<p><a href="https://news.ycombinator.com/item?id=42596883"><strong>Discuss this article on Hacker News</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-sg2000"><strong>My Current Project: “Apache NuttX RTOS for Sophgo SG2000”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-ox64"><strong>My Other Project: “NuttX for Ox64 BL808”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-star64"><strong>Older Project: “NuttX for Star64 JH7110”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx"><strong>Olderer Project: “NuttX for PinePhone”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io"><strong>Check out my articles</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/rss.xml"><strong>RSS Feed</strong></a></p>
</li>
</ul>
<p><em>Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…</em></p>
<p><a href="https://github.com/lupyuen/lupyuen.github.io/blob/master/src/bisect.md"><strong>lupyuen.org/src/bisect.md</strong></a></p>
<h1 id="appendix-inspect-executable-size-with-bloaty"><a class="doc-anchor" href="#appendix-inspect-executable-size-with-bloaty">§</a>10 Appendix: Inspect Executable Size with Bloaty</h1>
<p><em>Earlier we ran out of Stack Space. Has something grown too big?</em></p>
<p>We could run <strong>Bloaty</strong> to do detailed analysis of the <strong>Code and Data Size</strong>…</p>
<ul>
<li><a href="https://github.com/google/bloaty"><strong>github.com/google/bloaty</strong></a></li>
</ul>
<p>For quick experimenting: Bloaty is bundled inside our <strong>NuttX Docker Image</strong>…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Inside NuttX Docker:
## Assume we compiled NuttX at /root/nuttx/nuttx
$ /tools/bloaty/bin/bloaty /root/nuttx/nuttx
FILE SIZE VM SIZE
-------------- --------------
46.1% 6.80Mi 0.0% 0 .debug_info
17.1% 2.53Mi 0.0% 0 .debug_line
8.6% 1.26Mi 0.0% 0 .debug_abbrev
6.6% 1000Ki 0.0% 0 .debug_loclists
6.2% 941Ki 64.9% 941Ki .text
5.1% 772Ki 0.0% 0 .debug_str
2.5% 381Ki 26.3% 381Ki .rodata
1.8% 277Ki 0.0% 0 .debug_frame
1.7% 254Ki 0.0% 0 .symtab
1.2% 174Ki 0.0% 0 .strtab
1.1% 166Ki 0.0% 0 .debug_rnglists
1.1% 164Ki 0.0% 0 .debug_line_str
0.0% 0 8.1% 118Ki .bss
0.8% 114Ki 0.0% 0 .debug_aranges
0.1% 8.31Ki 0.6% 8.27Ki .data
0.0% 5.00Ki 0.1% 858 [104 Others]
0.0% 3.89Ki 0.0% 0 [Unmapped]
0.0% 2.97Ki 0.0% 0 .shstrtab
0.0% 296 0.0% 256 .srodata.cst8
0.0% 196 0.0% 0 [ELF Headers]
0.0% 144 0.0% 104 .sdata.called
100.0% 14.8Mi 100.0% 1.42Mi TOTAL
## Let&#39;s dump the details
## For NuttX QEMU RISC-V
$ /tools/bloaty/bin/bloaty \
/root/nuttx/nuttx \
-d compileunits
bloaty: Unknown ELF machine value: 243
## Oops Bloaty won&#39;t work for RISC-V Executable!</code></pre></div>
<p>Standard Bloaty won’t support RISC-V. But <a href="https://fuchsia.googlesource.com/third_party/bloaty/+/53360fd9826a417671a92386306745bfd5755f21%5E1..53360fd9826a417671a92386306745bfd5755f21/"><strong>Fuchsia Bloaty</strong></a> supports it.</p>
<p>We compile and run <strong>Fuchsia Bloaty</strong>…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Compile Fuchsia Bloaty for RISC-V Support
git clone https://fuchsia.googlesource.com/third_party/bloaty
cd bloaty
cmake -B build -G Ninja -S .
cmake --build build
## Run Fuchsia Bloaty on NuttX QEMU RISC-V
## Dump all the details
cd /root/nuttx
/root/bloaty/build/bloaty nuttx \
-d compileunits,segments,sections,symbols</code></pre></div>
<p>Now we see everything in our NuttX RISC-V Executable…</p>
<div class="example-wrap"><pre class="language-text"><code> FILE SIZE VM SIZE
-------------- --------------
62.7% 9.26Mi 66.2% 960Ki [2505 Others]
7.5% 1.11Mi 4.8% 69.2Ki EEE6secondB8un170006Ev
94.1% 1.04Mi 0.0% 0 [Unmapped]
55.6% 594Ki NAN% 0 .debug_info
21.3% 227Ki NAN% 0 .debug_line
17.4% 185Ki NAN% 0 .debug_str
3.1% 33.7Ki NAN% 0 .strtab
54.6% 18.4Ki NAN% 0 [160 Others]
7.8% 2.61Ki NAN% 0 std::__1::(anonymous namespace)::make&lt;&gt;()::buf
6.5% 2.20Ki NAN% 0 std::__1::num_get&lt;&gt;::do_get()
4.5% 1.52Ki NAN% 0 std::__1::num_put&lt;&gt;::do_put()</code></pre></div>
<p><a href="https://github.com/lupyuen/nuttx-bisect/releases/download/main-1/bloaty.log">(See the <strong>Complete Log</strong>)</a></p>
<!-- Begin scripts/rustdoc-after.html: Post-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker and Prism Theme -->
<script src="../theme.js"></script>
<script src="../prism.js"></script>
<!-- Theme Picker and Prism Theme -->
<!-- End scripts/rustdoc-after.html -->
</body>
</html>
Reference in a new issue View git blame Copy permalink