lupyuen.org/articles/usb3.html

995 lines
No EOL
59 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="rustdoc">
<title>NuttX RTOS for PinePhone: Simpler USB with EHCI (Enhanced Host Controller Interface)</title>
<!-- Begin scripts/articles/*-header.html: Article Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<meta property="og:title"
content="NuttX RTOS for PinePhone: Simpler USB with EHCI (Enhanced Host Controller Interface)"
data-rh="true">
<meta property="og:description"
content="Porting the Enhanced Host Controller Interface (EHCI) USB Driver... From Apache NuttX RTOS to Pine64 PinePhone"
data-rh="true">
<meta name="description"
content="Porting the Enhanced Host Controller Interface (EHCI) USB Driver... From Apache NuttX RTOS to Pine64 PinePhone">
<meta property="og:image"
content="https://lupyuen.github.io/images/usb3-title.jpg">
<meta property="og:type"
content="article" data-rh="true">
<link rel="canonical" href="https://lupyuen.org/articles/usb3.html" />
<!-- End scripts/articles/*-header.html -->
<!-- Begin scripts/rustdoc-header.html: Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<link rel="alternate" type="application/rss+xml" title="RSS Feed for lupyuen" href="/rss.xml" />
<link rel="stylesheet" type="text/css" href="../normalize.css">
<link rel="stylesheet" type="text/css" href="../rustdoc.css" id="mainThemeStyle">
<link rel="stylesheet" type="text/css" href="../dark.css">
<link rel="stylesheet" type="text/css" href="../light.css" id="themeStyle">
<link rel="stylesheet" type="text/css" href="../prism.css">
<script src="../storage.js"></script><noscript>
<link rel="stylesheet" href="../noscript.css"></noscript>
<link rel="shortcut icon" href="../favicon.ico">
<style type="text/css">
#crate-search {
background-image: url("../down-arrow.svg");
}
</style>
<!-- End scripts/rustdoc-header.html -->
</head>
<body class="rustdoc">
<!--[if lte IE 8]>
<div class="warning">
This old browser is unsupported and will most likely display funky
things.
</div>
<![endif]-->
<!-- Begin scripts/rustdoc-before.html: Pre-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker -->
<div class="theme-picker" style="left: 0"><button id="theme-picker" aria-label="Pick another theme!"><img src="../brush.svg"
width="18" alt="Pick another theme!"></button>
<div id="theme-choices"></div>
</div>
<!-- Theme Picker -->
<!-- End scripts/rustdoc-before.html -->
<h1 class="title">NuttX RTOS for PinePhone: Simpler USB with EHCI (Enhanced Host Controller Interface)</h1>
<nav id="rustdoc"><ul>
<li><a href="#usb-enhanced-host-controller-interface" title="USB Enhanced Host Controller Interface">1 USB Enhanced Host Controller Interface</a><ul></ul></li>
<li><a href="#ehci-is-simpler-than-usb-on-the-go" title="EHCI is simpler than USB On-The-Go">2 EHCI is simpler than USB On-The-Go</a><ul></ul></li>
<li><a href="#pinephone-usb-controller" title="PinePhone USB Controller">3 PinePhone USB Controller</a><ul></ul></li>
<li><a href="#ehci-driver-from-apache-nuttx" title="EHCI Driver from Apache NuttX">4 EHCI Driver from Apache NuttX</a><ul></ul></li>
<li><a href="#64-bit-update-for-ehci-driver" title="64-Bit Update for EHCI Driver">5 64-Bit Update for EHCI Driver</a><ul></ul></li>
<li><a href="#halt-timeout-for-usb-controller" title="Halt Timeout for USB Controller">6 Halt Timeout for USB Controller</a><ul></ul></li>
<li><a href="#pinephone-usb-drivers-in-u-boot-bootloader" title="PinePhone USB Drivers in U-Boot Bootloader">7 PinePhone USB Drivers in U-Boot Bootloader</a><ul></ul></li>
<li><a href="#power-on-the-usb-controller" title="Power On the USB Controller">8 Power On the USB Controller</a><ul></ul></li>
<li><a href="#enable-usb-controller-clocks" title="Enable USB Controller Clocks">9 Enable USB Controller Clocks</a><ul></ul></li>
<li><a href="#reset-usb-controller" title="Reset USB Controller">10 Reset USB Controller</a><ul></ul></li>
<li><a href="#nuttx-ehci-driver-starts-ok-on-pinephone" title="NuttX EHCI Driver Starts OK on PinePhone">11 NuttX EHCI Driver Starts OK on PinePhone</a><ul></ul></li>
<li><a href="#whats-next" title="Whats Next">12 Whats Next</a><ul></ul></li></ul></nav><p>📝 <em>24 Mar 2023</em></p>
<p><img src="https://lupyuen.github.io/images/usb3-title.jpg" alt="USB Controller Block Diagram from Allwinner A64 User Manual" /></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx/releases/download/doc/Allwinner_A64_User_Manual_V1.1.pdf"><em>USB Controller Block Diagram from Allwinner A64 User Manual</em></a></p>
<p>Weeks ago we talked about porting <a href="https://lupyuen.github.io/articles/what"><strong>Apache NuttX RTOS</strong></a> (Real-Time Operating System) to <a href="https://wiki.pine64.org/index.php/PinePhone"><strong>Pine64 PinePhone</strong></a>. And how we might turn it into a <strong>Feature Phone</strong></p>
<ul>
<li><a href="https://lupyuen.github.io/articles/usb2#pinephone--nuttx--feature-phone"><strong>“PinePhone + NuttX = Feature Phone”</strong></a></li>
</ul>
<p>But to make phone calls and send text messages, we need to control the <strong>LTE Modem over USB</strong></p>
<ul>
<li><a href="https://lupyuen.github.io/articles/usb2#lte-modem-talks-usb"><strong>“LTE Modem talks USB”</strong></a></li>
</ul>
<p>Thus today well build a <strong>USB Driver</strong> for NuttX on PinePhone. As we find out…</p>
<ul>
<li>
<p>Whats <strong>USB Enhanced Host Controller Interface</strong> (EHCI)</p>
</li>
<li>
<p>Why its simpler than <strong>USB On-The-Go</strong> (OTG)</p>
</li>
<li>
<p>How we ported the <strong>USB EHCI Driver</strong> from NuttX to PinePhone</p>
</li>
<li>
<p>Handling <strong>USB Clocks</strong> and <strong>USB Resets</strong> on PinePhone</p>
<p>(Based on tips from <strong>U-Boot Bootloader</strong>)</p>
</li>
<li>
<p>And the NuttX EHCI Driver <strong>boots OK on PinePhone!</strong> 🎉</p>
</li>
</ul>
<p>Lets dive into the fascinating world of USB EHCI…</p>
<p><a href="https://lupyuen.github.io/articles/usb2#appendix-enhanced-host-controller-interface-for-usb">(Thanks to <strong>Lwazi Dube</strong> for teaching me about EHCI 🙂)</a></p>
<p><img src="https://lupyuen.github.io/images/usb2-ehci3.jpg" alt="USB EHCI Registers in Allwinner A64 User Manual (Page 585)" /></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx/releases/download/doc/Allwinner_A64_User_Manual_V1.1.pdf"><em>USB EHCI Registers in Allwinner A64 User Manual (Page 585)</em></a></p>
<h1 id="usb-enhanced-host-controller-interface"><a class="doc-anchor" href="#usb-enhanced-host-controller-interface">§</a>1 USB Enhanced Host Controller Interface</h1>
<p><em>Whats USB EHCI?</em></p>
<p>According to the <a href="https://www.intel.sg/content/www/xa/en/products/docs/io/universal-serial-bus/ehci-specification.html"><strong>Official Spec</strong></a></p>
<blockquote>
<p>“The <strong>Enhanced Host Controller Interface (EHCI)</strong> specification describes the <strong>Register-Level Interface</strong> for a Host Controller for the Universal Serial Bus (USB) Revision 2.0”</p>
</blockquote>
<blockquote>
<p>“The specification includes a description of the Hardware and Software Interface between System Software and the Host Controller Hardware”</p>
</blockquote>
<p><em>So EHCI is a standard, unified way to program the USB Controller on any Hardware Platform?</em></p>
<p>Yep and USB EHCI is <strong>supported on PinePhone</strong>!</p>
<p>Which means we can build the USB Driver for PinePhone… By simply reading and writing the (Memory-Mapped) <strong>EHCI Registers</strong> in Allwinner A64 USB Controller! (Pic above)</p>
<p><em>What are the USB EHCI Registers?</em></p>
<p>The <strong>Standard EHCI Registers</strong> are documented here…</p>
<ul>
<li>
<p><a href="https://www.intel.sg/content/www/xa/en/products/docs/io/universal-serial-bus/ehci-specification.html"><strong>“Enhanced Host Controller Interface Specification”</strong></a></p>
</li>
<li>
<p><a href="https://www.intel.sg/content/www/xa/en/products/docs/io/universal-serial-bus/ehci-specification-for-usb.html"><strong>“Enhanced Host Controller Interface for USB 2.0: Specification”</strong></a></p>
<p>(Skip the “Version 1.1 Addendum”, Allwinner A64 only implements Version 1.0 of the spec)</p>
</li>
</ul>
<p>Allwinner A64 implements the EHCI Registers for <strong>Port USB1</strong> at…</p>
<ul>
<li>
<p><strong>USB_HCI1</strong> Base Address: <strong><code>0x01C1</code> <code>B000</code></strong></p>
<p>(Pic above)</p>
</li>
</ul>
<p>More about this in the <a href="https://github.com/lupyuen/pinephone-nuttx/releases/download/doc/Allwinner_A64_User_Manual_V1.1.pdf"><strong>Allwinner A64 User Manual</strong></a></p>
<ul>
<li>
<p><strong>Section 7.5.3.3:</strong> USB Host Register List (Page 585, pic above)</p>
</li>
<li>
<p><strong>Section 7.5.3.4:</strong> EHCI Register Description (Page 587)</p>
</li>
<li>
<p><strong>Section 7.5.3.5:</strong> OHCI Register Description (Page 601)</p>
</li>
<li>
<p><strong>Section 7.5.3.6:</strong> HCI Interface Control and Status Register Description (Page 619)</p>
</li>
<li>
<p><strong>Section 7.5.3.7:</strong> USB Host Clock Requirement (Page 620)</p>
</li>
</ul>
<p>This looks messy, but the <strong>NuttX EHCI Driver</strong> will probably run OK on PinePhone.</p>
<p><em>USB EHCI sounds like a lifesaver?</em></p>
<p>Yep USB Programming on PinePhone would be super complicated without EHCI!</p>
<p>Lets take a peek at life without EHCI…</p>
<p><a href="https://en.wikipedia.org/wiki/Human%E2%80%93computer_interaction">(EHCI always reminds me of Don Norman)</a></p>
<p><img src="https://lupyuen.github.io/images/arm-uart2.jpg" alt="PinePhone Jumpdrive appears as a USB Drive when connected to a computer" /></p>
<p><a href="https://github.com/dreemurrs-embedded/Jumpdrive"><em>PinePhone Jumpdrive appears as a USB Drive when connected to a computer</em></a></p>
<h1 id="ehci-is-simpler-than-usb-on-the-go"><a class="doc-anchor" href="#ehci-is-simpler-than-usb-on-the-go">§</a>2 EHCI is simpler than USB On-The-Go</h1>
<p><em>Whats USB On-The-Go?</em></p>
<p>PinePhone supports <a href="https://en.wikipedia.org/wiki/USB_On-The-Go"><strong>USB On-The-Go (OTG)</strong></a>, which works as two modes…</p>
<ul>
<li>
<p><strong>USB Host</strong>: PinePhone controls other USB Devices</p>
</li>
<li>
<p><strong>USB Device</strong>: PinePhone is controlled by a USB Host</p>
</li>
</ul>
<p>This means if we connect PinePhone to a computer, it will appear as a USB Drive.</p>
<p>(Assuming the right drivers are started)</p>
<p><em>Is USB OTG compatible with USB EHCI?</em></p>
<p>EHCI supports <strong>only USB Host</strong>, not USB Device.</p>
<p>(Hence the name “Enhanced <strong>Host Controller</strong> Interface”)</p>
<p>PinePhone supports both USB OTG and USB EHCI. PinePhones USB Physical Layer can switch between OTG and EHCI modes. (As well soon see)</p>
<p><em>How would we program USB OTG?</em></p>
<p>To do USB OTG, we would need to create a driver for the <strong>Mentor Graphics (MUSB) OTG Controller</strong> inside PinePhone…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/usb2#document-the-usb-controller"><strong>“Document the USB Controller (Mentor Graphics)”</strong></a></li>
</ul>
<p>Which gets really low-level and complex. <a href="https://lupyuen.github.io/articles/usb2#stm32-usb-driver-for-nuttx">(Like this)</a></p>
<p>Thankfully we wont need USB OTG and the Mentor Graphics Driver. Heres why…</p>
<p><img src="https://lupyuen.github.io/images/usb3-title.jpg" alt="USB Controller Block Diagram from Allwinner A64 User Manual (Page 583)" /></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx/releases/download/doc/Allwinner_A64_User_Manual_V1.1.pdf"><em>USB Controller Block Diagram from Allwinner A64 User Manual (Page 583)</em></a></p>
<h1 id="pinephone-usb-controller"><a class="doc-anchor" href="#pinephone-usb-controller">§</a>3 PinePhone USB Controller</h1>
<p><em>Phew! Were doing USB EHCI, not USB OTG?</em></p>
<p>According to the <a href="https://github.com/lupyuen/pinephone-nuttx/releases/download/doc/Allwinner_A64_User_Manual_V1.1.pdf"><strong>Allwinner A64 User Manual</strong></a> (Page 583), there are two USB Ports in Allwinner A64: <strong>USB0 and USB1</strong></p>
<ul>
<li>
<p><strong>Port USB0</strong> is exposed as the <strong>External USB Port</strong> on PinePhone</p>
<p>(Top part of pic above)</p>
</li>
<li>
<p><strong>Port USB1</strong> is connected to the <strong>Internal LTE Modem</strong></p>
<p>(Bottom part of pic above)</p>
</li>
</ul>
<p>The names are kinda confusing in the A64 User Manual…</p>
<div><table><thead><tr><th style="text-align: center">USB Port</th><th>Alternate Name</th><th>Base Address</th></tr></thead><tbody>
<tr><td style="text-align: center"><strong>Port USB0</strong></td><td>USB-OTG-EHCI / OHCI</td><td><strong><code>0x01C1</code> <code>A000</code></strong> (USB_HCI0)</td></tr>
<tr><td style="text-align: center"><strong>Port USB1</strong></td><td>USB-EHCI0 / OHCI0</td><td><strong><code>0x01C1</code> <code>B000</code></strong> (USB_HCI1)</td></tr>
</tbody></table>
</div>
<p>Port USB0 isnt documented, but it appears in the <strong>Memory Mapping</strong> of <a href="https://github.com/lupyuen/pinephone-nuttx/releases/download/doc/Allwinner_A64_User_Manual_V1.1.pdf"><strong>Allwinner A64 User Manual</strong></a>. (Page 73)</p>
<p><em>But they look so different in the pic…</em></p>
<p>They aint two peas in a pod of pink dolphins because…</p>
<ul>
<li>
<p>Only <strong>Port USB0</strong> supports <a href="https://lupyuen.github.io/articles/usb3#ehci-is-simpler-than-usb-on-the-go"><strong>USB On-The-Go (OTG)</strong></a>.</p>
<p>Which means if we connect PinePhone to a computer, it will appear as a USB Drive. (Assuming the right drivers are started)</p>
<p>Thats why Port USB0 is exposed as the <strong>External USB Port</strong> on PinePhone.</p>
</li>
<li>
<p>Both <strong>USB0 and USB1</strong> support <a href="https://lupyuen.github.io/articles/usb3#usb-enhanced-host-controller-interface"><strong>USB Enhanced Host Controller Interface (EHCI)</strong></a>.</p>
<p>Which will work only as a USB Host. (Not USB Device)</p>
<p>And thats perfectly hunky dory for the <strong>LTE Modem</strong> on USB1. (Pic below)</p>
</li>
</ul>
<p><em>We need the LTE Modem for our Feature Phone?</em></p>
<p>Exactly! Today were making a <a href="https://lupyuen.github.io/articles/usb2#pinephone--nuttx--feature-phone"><strong>Feature Phone</strong></a> with the <a href="https://lupyuen.github.io/articles/usb2#lte-modem-talks-usb"><strong>LTE Modem</strong></a>.</p>
<p>So well talk only about <strong>Port USB1</strong> (EHCI / Non-OTG), since its connected to the LTE Modem. (Pic below)</p>
<p>Lets build the EHCI Driver…</p>
<p><img src="https://lupyuen.github.io/images/usb2-title.jpg" alt="Quectel EG25-G LTE Modem in PinePhone Schematic (Page 15)" /></p>
<p><a href="https://files.pine64.org/doc/PinePhone/PinePhone%20v1.2b%20Released%20Schematic.pdf"><em>Quectel EG25-G LTE Modem in PinePhone Schematic (Page 15)</em></a></p>
<h1 id="ehci-driver-from-apache-nuttx"><a class="doc-anchor" href="#ehci-driver-from-apache-nuttx">§</a>4 EHCI Driver from Apache NuttX</h1>
<p><em>Does NuttX have a USB EHCI Driver?</em></p>
<p>Yep! Apache NuttX RTOS has a <strong>USB EHCI Driver</strong></p>
<ul>
<li>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/arm/src/imxrt/imxrt_ehci.c#L4970"><strong>NuttX EHCI Driver (NXP i.MX RT)</strong></a></p>
<p><a href="https://lupyuen.github.io/articles/usb2#appendix-enhanced-host-controller-interface-for-usb">(Other EHCI Drivers in NuttX are similar)</a></p>
</li>
</ul>
<p>Which well <strong>port to PinePhone</strong> as…</p>
<ul>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb"><strong>PinePhone USB Driver for NuttX</strong></a></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb#pinephone-usb-driver-for-apache-nuttx-rtos">(Interim Build Instructions)</a></p>
</li>
</ul>
<p><em>But the EHCI Register Addresses are specific to PinePhone right?</em></p>
<p>Thats why we customised the <strong>EHCI Register Addresses</strong> specially for PinePhone and Allwinner A64: <a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/2f6c49aafbaa3b15f47107af19c92eaa92eac2e1/a64_usbotg.h#L40-L55">a64_usbotg.h</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Address of EHCI Device / Host Capability Registers
// For Allwinner A64: USB_HCI1
#define A64_USBOTG_HCCR_BASE 0x01c1b000
// Address of Device / Host / OTG Operational Registers
// For Allwinner A64: USB_HCI1 + 0x10
#define A64_USBOTG_HCOR_BASE (A64_USBOTG_HCCR_BASE + 0x10)</code></pre></div>
<p><a href="https://lupyuen.github.io/articles/usb3#usb-enhanced-host-controller-interface">(EHCI Base Address is <strong><code>0x01C1</code> <code>B000</code></strong>)</a></p>
<p>We start the USB EHCI Driver in the <strong>PinePhone Bringup Function</strong>: <a href="https://github.com/lupyuen2/wip-nuttx/blob/usb/boards/arm64/a64/pinephone/src/pinephone_bringup.c#L208-L213">pinephone_bringup.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>int pinephone_bringup(void) {
...
// Start the USB EHCI Driver
ret = a64_usbhost_initialize();</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/main/a64_usbhost.c#L260-L364">(<strong>a64_usbhost_initialize</strong> is defined here)</a></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/main/a64_ehci.c#L4953-L5373">(Which calls <strong>a64_ehci_initialize</strong>)</a></p>
<p>Lets boot our new EHCI Driver on PinePhone and watch what happens…</p>
<h1 id="64-bit-update-for-ehci-driver"><a class="doc-anchor" href="#64-bit-update-for-ehci-driver">§</a>5 64-Bit Update for EHCI Driver</h1>
<p><em>What happens when we boot NuttX with our customised EHCI Driver?</em></p>
<p>When NuttX boots our EHCI Driver for PinePhone, it halts with an <strong>Assertion Failure</strong></p>
<div class="example-wrap"><pre class="language-text"><code>Assertion failed:
at file: chip/a64_ehci.c:4996
task: nsh_main 0x4008b0d0</code></pre></div>
<p>Which says that the <strong>a64_qh_s</strong> struct must be <strong>aligned to 32 bytes</strong>: <a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/b80499b3b8ec837fe2110e9476e8a6ad0f194cde/a64_ehci.c#L4996">a64_ehci.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>DEBUGASSERT((sizeof(struct a64_qh_s) &amp; 0x1f) == 0);</code></pre></div>
<p>But somehow its not! The actual size of the <strong>a64_qh_s</strong> struct is <strong>72 bytes</strong></p>
<div class="example-wrap"><pre class="language-text"><code>sizeof(struct a64_qh_s) = 72</code></pre></div>
<p>Which most certainly <strong>isnt aligned</strong> to 32 bytes.</p>
<p><em>Huh? Whats with the struct size?</em></p>
<p>Take a guess! Heres the definition of <strong>a64_qh_s</strong>: <a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/b80499b3b8ec837fe2110e9476e8a6ad0f194cde/a64_ehci.c#L186-L200">a64_ehci.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Internal representation of the EHCI Queue Head (QH)
struct a64_qh_s {
// Hardware representation of the queue (head)
struct ehci_qh_s hw;
// Endpoint used for the transfer
struct a64_epinfo_s *epinfo;
// First qTD in the list (physical address)
uint32_t fqp;
// Padding to assure 32-byte alignment
uint8_t pad[8];
};</code></pre></div>
<p><em>The pointer looks sus…</em></p>
<p>Yep <strong>epinfo</strong> is a <strong>pointer</strong>, normally <strong>4 bytes</strong> on 32-bit platforms…</p>
<div class="example-wrap"><pre class="language-c"><code> // Pointer Size is Platform Dependent
struct a64_epinfo_s *epinfo;</code></pre></div>
<p>But PinePhone is the very first <strong>Arm64 port</strong> of NuttX!</p>
<p>Thus <strong>epinfo</strong> actually occupies <strong>8 bytes</strong> on PinePhone and other 64-bit platforms.</p>
<p><em>How has the struct changed for 32-bit platforms vs 64-bit platforms?</em></p>
<ul>
<li>
<p>On <strong>32-bit</strong> platforms: <strong>a64_qh_s</strong> was previously <strong>64 bytes</strong></p>
<p>(48 + 4 + 4 + 8)</p>
</li>
<li>
<p>On <strong>64-bit</strong> platforms: <strong>a64_qh_s</strong> is now <strong>72 bytes</strong></p>
<p>(48 + 8 + 4 + 8, round up for 4-byte alignment)</p>
</li>
</ul>
<p>We fix this by padding <strong>a64_qh_s</strong> from 72 bytes to 96 bytes…</p>
<div class="example-wrap"><pre class="language-c"><code>struct a64_qh_s {
...
// Original Padding: 8 bytes
uint8_t pad[8];
// Added this: Pad from 72 to 96 bytes for 64-bit platforms
uint8_t pad2[96 - 72];
};</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/2e1f9ab090b14f88afb8c3a36ec40a0dbbb23d49/a64_ehci.c#L190-L202">(Source)</a></p>
<p>And this fixes our Assertion Failure!</p>
<p><em>This 64-bit patching sounds scary… What about other structs?</em></p>
<p>To be safe, we verified that the other Struct Sizes are still <strong>valid for 64-bit platforms</strong>: <a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/2e1f9ab090b14f88afb8c3a36ec40a0dbbb23d49/a64_ehci.c#L4999-L5004">a64_ehci.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>DEBUGASSERT(sizeof(struct ehci_itd_s) == SIZEOF_EHCI_ITD_S);
DEBUGASSERT(sizeof(struct ehci_sitd_s) == SIZEOF_EHCI_SITD_S);
DEBUGASSERT(sizeof(struct ehci_qtd_s) == SIZEOF_EHCI_QTD_S);
DEBUGASSERT(sizeof(struct ehci_overlay_s) == 32);
DEBUGASSERT(sizeof(struct ehci_qh_s) == 48);
DEBUGASSERT(sizeof(struct ehci_fstn_s) == SIZEOF_EHCI_FSTN_S);</code></pre></div>
<p>FYI: These are the <strong>Struct Sizes</strong> in the EHCI Driver…</p>
<div class="example-wrap"><pre class="language-text"><code>sizeof(struct a64_qh_s) = 72
sizeof(struct a64_qtd_s) = 32
sizeof(struct ehci_itd_s) = 64
sizeof(struct ehci_sitd_s) = 28
sizeof(struct ehci_qtd_s) = 32
sizeof(struct ehci_overlay_s) = 32
sizeof(struct ehci_qh_s) = 48
sizeof(struct ehci_fstn_s) = 8</code></pre></div>
<p>Lets continue booting NuttX…</p>
<p><a href="https://github.com/apache/nuttx/blob/master/include/nuttx/usb/ehci.h#L955-L974">(We need to fix this NuttX typo: <strong>SIZEOF_EHCI_OVERLAY</strong> is defined twice)</a></p>
<h1 id="halt-timeout-for-usb-controller"><a class="doc-anchor" href="#halt-timeout-for-usb-controller">§</a>6 Halt Timeout for USB Controller</h1>
<p><em>So NuttX boots without an Assertion Failure?</em></p>
<p>Yeah but our USB EHCI Driver <strong>fails with a timeout</strong> when booting on PinePhone…</p>
<div class="example-wrap"><pre class="language-text"><code>usbhost_registerclass:
Registering class:0x40124838 nids:2
EHCI Initializing EHCI Stack
a64_printreg:
01c1b010&lt;-00000000
a64_printreg:
01c1b014-&gt;00000000
EHCI ERROR:
Timed out waiting for HCHalted.
USBSTS: 000000
EHCI ERROR:
a64_reset failed: 110
a64_usbhost_initialize:
ERROR: a64_ehci_initialize failed</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/b921aa5259ef94ece41610ebf806ebd0fa19dee5/README.md#output-log">(Source)</a></p>
<p>The timeout happens while waiting for the <strong>USB Controller to Halt</strong>: <a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/2e1f9ab090b14f88afb8c3a36ec40a0dbbb23d49/a64_ehci.c#L4831-L4917">a64_ehci.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Reset the USB EHCI Controller
static int a64_reset(void) {
// Halt the EHCI Controller
a64_putreg(0, &amp;HCOR-&gt;usbcmd);
// Wait for EHCI Controller to halt
timeout = 0;
do {
// Wait one microsecond and update the timeout counter
up_udelay(1); timeout++;
// Get the current value of the USBSTS register
regval = a64_getreg(&amp;HCOR-&gt;usbsts);
}
while (((regval &amp; EHCI_USBSTS_HALTED) == 0) &amp;&amp; (timeout &lt; 1000));
// Is the EHCI still running? Did we timeout?
if ((regval &amp; EHCI_USBSTS_HALTED) == 0) {
// Here&#39;s the Halt Timeout that we hit
usbhost_trace1(EHCI_TRACE1_HCHALTED_TIMEOUT, regval);
return -ETIMEDOUT;
}</code></pre></div>
<p><em>Whats a64_putreg and a64_getreg?</em></p>
<p>Our EHCI Driver calls <a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/2e1f9ab090b14f88afb8c3a36ec40a0dbbb23d49/a64_ehci.c#L964-L989"><strong>a64_getreg</strong></a> and <a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/2e1f9ab090b14f88afb8c3a36ec40a0dbbb23d49/a64_ehci.c#L991-L1015"><strong>a64_putreg</strong></a> to read and write the <a href="https://lupyuen.github.io/articles/usb3#usb-enhanced-host-controller-interface"><strong>EHCI Registers</strong></a>.</p>
<p>They appear in our log like so…</p>
<div class="example-wrap"><pre class="language-text"><code>a64_printreg:
01c1b010&lt;-00000000
a64_printreg:
01c1b014-&gt;00000000</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/b921aa5259ef94ece41610ebf806ebd0fa19dee5/README.md#output-log">(Source)</a></p>
<p>Which means that our driver has written 0 to <code>01C1</code> <code>B010</code>, and read 0 from <code>01C1</code> <code>B014</code>.</p>
<p><em>What are 01C1 B010 and 01C1 B014?</em></p>
<ul>
<li>
<p><strong><code>01C1</code> <code>B000</code></strong> is the Base Address of the <strong>USB EHCI Controller</strong> on Allwinner A64</p>
<p><a href="https://lupyuen.github.io/articles/usb3#usb-enhanced-host-controller-interface">(See this)</a></p>
</li>
<li>
<p><strong><code>01C1</code> <code>B010</code></strong> is the <strong>USB Command Register USBCMD</strong></p>
<p><a href="https://www.intel.sg/content/www/xa/en/products/docs/io/universal-serial-bus/ehci-specification-for-usb.html">(EHCI Spec, Page 18)</a></p>
</li>
<li>
<p><strong><code>01C1</code> <code>B014</code></strong> is the <strong>USB Status Register USBSTS</strong></p>
<p><a href="https://www.intel.sg/content/www/xa/en/products/docs/io/universal-serial-bus/ehci-specification-for-usb.html">(EHCI Spec, Page 21)</a></p>
</li>
</ul>
<p>When we see this…</p>
<div class="example-wrap"><pre class="language-text"><code>a64_printreg:
01c1b010&lt;-00000000
a64_printreg:
01c1b014-&gt;00000000</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/b921aa5259ef94ece41610ebf806ebd0fa19dee5/README.md#output-log">(Source)</a></p>
<p>It means…</p>
<ol>
<li>
<p>Our driver wrote <strong>Command 0 (Stop)</strong> to <strong>USB Command Register USBCMD</strong>.</p>
<p>Which should Halt the USB Controller.</p>
</li>
<li>
<p>Then we read <strong>USB Status Register USBSTS</strong>.</p>
<p>This returns 0, which means that the USB Controller <strong>has NOT been Halted</strong>.</p>
<p>(HCHalted = 0)</p>
</li>
</ol>
<p>Thats why the USB Driver failed: It <strong>couldnt Halt the USB Controller</strong> at startup.</p>
<p><em>Why?</em></p>
<p>Probably because we <strong>havent powered on</strong> the USB Controller? Says our log…</p>
<div class="example-wrap"><pre class="language-text"><code>TODO: Switch off USB bus power
TODO: Setup pins, with power initially off
TODO: Reset the controller from the OTG peripheral
TODO: Program the controller to be the USB host controller</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/b921aa5259ef94ece41610ebf806ebd0fa19dee5/README.md#output-log">(Source)</a></p>
<p>And maybe we need to initialise the <strong>USB Physical Layer</strong>.</p>
<p><em>How do we power on the USB Controller?</em></p>
<p>Lets get inspired by consulting the U-Boot Bootloader…</p>
<p><img src="https://lupyuen.github.io/images/uboot-uboot.png" alt="U-Boot Bootloader on PinePhone" /></p>
<p><a href="https://lupyuen.github.io/articles/uboot#u-boot-bootloader"><em>U-Boot Bootloader on PinePhone</em></a></p>
<h1 id="pinephone-usb-drivers-in-u-boot-bootloader"><a class="doc-anchor" href="#pinephone-usb-drivers-in-u-boot-bootloader">§</a>7 PinePhone USB Drivers in U-Boot Bootloader</h1>
<p><em>We need to power on PinePhones USB Controller…</em></p>
<p><em>How can U-Boot Bootloader help?</em></p>
<p><a href="https://lupyuen.github.io/articles/uboot#u-boot-bootloader"><strong>U-Boot Bootloader</strong></a> is the very first thing that runs when we power on our PinePhone.</p>
<p>U-Boot allows booting from a USB Drive… Thus it must have a <strong>USB Driver inside</strong>!</p>
<p>Lets find the PinePhone USB Driver inside U-Boot and study it.</p>
<p><em>How to find the PinePhone USB Driver in U-Boot?</em></p>
<p>When we search for PinePhone in the Source Code of <a href="https://github.com/u-boot/u-boot"><strong>U-Boot Bootloader</strong></a>, we find this Build Configuration: <a href="https://github.com/u-boot/u-boot/blob/master/configs/pinephone_defconfig#L3">pinephone_defconfig</a></p>
<div class="example-wrap"><pre class="language-text"><code>CONFIG_DEFAULT_DEVICE_TREE=&quot;sun50i-a64-pinephone-1.2&quot;</code></pre></div>
<p>Which refers to this <strong>PinePhone Device Tree</strong>: <a href="https://github.com/u-boot/u-boot/blob/master/arch/arm/dts/sun50i-a64-pinephone-1.2.dts#L6">sun50i-a64-pinephone-1.2.dts</a></p>
<div class="example-wrap"><pre class="language-text"><code>#include &quot;sun50i-a64-pinephone.dtsi&quot;</code></pre></div>
<p>Which includes <strong>another Device Tree</strong>: <a href="https://github.com/u-boot/u-boot/blob/master/arch/arm/dts/sun50i-a64-pinephone.dtsi#L153-L516">sun50i-a64-pinephone.dtsi</a></p>
<div class="example-wrap"><pre class="language-text"><code>#include &quot;sun50i-a64.dtsi&quot;
#include &quot;sun50i-a64-cpu-opp.dtsi&quot;
...
&amp;ehci0 { status = &quot;okay&quot;; };
&amp;ehci1 { status = &quot;okay&quot;; };
&amp;usb_otg {
dr_mode = &quot;peripheral&quot;;
status = &quot;okay&quot;;
};
&amp;usb_power_supply { status = &quot;okay&quot;; };
&amp;usbphy { status = &quot;okay&quot;; };</code></pre></div>
<p>Which includes this <strong>Allwinner A64 Device Tree</strong>: <a href="https://github.com/u-boot/u-boot/blob/master/arch/arm/dts/sun50i-a64.dtsi#L575-L587">sun50i-a64.dtsi</a></p>
<div class="example-wrap"><pre class="language-text"><code>usb_otg: usb@1c19000 {
compatible = &quot;allwinner,sun8i-a33-musb&quot;;
reg = &lt;0x01c19000 0x0400&gt;;
clocks = &lt;&amp;ccu CLK_BUS_OTG&gt;;
resets = &lt;&amp;ccu RST_BUS_OTG&gt;;
interrupts = &lt;GIC_SPI 71 IRQ_TYPE_LEVEL_HIGH&gt;;
interrupt-names = &quot;mc&quot;;
phys = &lt;&amp;usbphy 0&gt;;
phy-names = &quot;usb&quot;;
extcon = &lt;&amp;usbphy 0&gt;;
dr_mode = &quot;otg&quot;;
status = &quot;disabled&quot;;
};</code></pre></div>
<p>Thats for <a href="https://lupyuen.github.io/articles/usb3#ehci-is-simpler-than-usb-on-the-go"><strong>USB OTG (On-The-Go)</strong></a>, which well skip today.</p>
<p>Next comes the <strong>USB PHY (Physical Layer)</strong>, which is the electrical wiring for Ports USB0 and USB1: <a href="https://github.com/u-boot/u-boot/blob/master/arch/arm/dts/sun50i-a64.dtsi#L589-L607">sun50i-a64.dtsi</a></p>
<div class="example-wrap"><pre class="language-text"><code>usbphy: phy@1c19400 {
compatible = &quot;allwinner,sun50i-a64-usb-phy&quot;;
reg =
&lt;0x01c19400 0x14&gt;,
&lt;0x01c1a800 0x4&gt;,
&lt;0x01c1b800 0x4&gt;;
reg-names =
&quot;phy_ctrl&quot;,
&quot;pmu0&quot;,
&quot;pmu1&quot;;
clocks =
&lt;&amp;ccu CLK_USB_PHY0&gt;,
&lt;&amp;ccu CLK_USB_PHY1&gt;;
clock-names =
&quot;usb0_phy&quot;,
&quot;usb1_phy&quot;;
resets =
&lt;&amp;ccu RST_USB_PHY0&gt;,
&lt;&amp;ccu RST_USB_PHY1&gt;;
reset-names =
&quot;usb0_reset&quot;,
&quot;usb1_reset&quot;;
status = &quot;disabled&quot;;
#phy-cells = &lt;1&gt;;
};</code></pre></div>
<p>(Well come back to <strong>clocks</strong> and <strong>resets</strong> in a while)</p>
<p>Then comes the <strong>EHCI Controller</strong> for <strong>Port USB0</strong> (which well skip): <a href="https://github.com/u-boot/u-boot/blob/master/arch/arm/dts/sun50i-a64.dtsi#L609-L633">sun50i-a64.dtsi</a></p>
<div class="example-wrap"><pre class="language-text"><code>ehci0: usb@1c1a000 {
compatible = &quot;allwinner,sun50i-a64-ehci&quot;, &quot;generic-ehci&quot;;
reg = &lt;0x01c1a000 0x100&gt;;
interrupts = &lt;GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH&gt;;
clocks =
&lt;&amp;ccu CLK_BUS_OHCI0&gt;,
&lt;&amp;ccu CLK_BUS_EHCI0&gt;,
&lt;&amp;ccu CLK_USB_OHCI0&gt;;
resets =
&lt;&amp;ccu RST_BUS_OHCI0&gt;,
&lt;&amp;ccu RST_BUS_EHCI0&gt;;
phys = &lt;&amp;usbphy 0&gt;;
phy-names = &quot;usb&quot;;
status = &quot;disabled&quot;;
};</code></pre></div>
<p>Finally the <strong>EHCI Controller</strong> for <strong>Port USB1</strong> (which we need): <a href="https://github.com/u-boot/u-boot/blob/master/arch/arm/dts/sun50i-a64.dtsi#L635-L659">sun50i-a64.dtsi</a></p>
<div class="example-wrap"><pre class="language-text"><code>ehci1: usb@1c1b000 {
compatible = &quot;allwinner,sun50i-a64-ehci&quot;, &quot;generic-ehci&quot;;
reg = &lt;0x01c1b000 0x100&gt;;
interrupts = &lt;GIC_SPI 74 IRQ_TYPE_LEVEL_HIGH&gt;;
clocks =
&lt;&amp;ccu CLK_BUS_OHCI1&gt;,
&lt;&amp;ccu CLK_BUS_EHCI1&gt;,
&lt;&amp;ccu CLK_USB_OHCI1&gt;;
resets =
&lt;&amp;ccu RST_BUS_OHCI1&gt;,
&lt;&amp;ccu RST_BUS_EHCI1&gt;;
phys = &lt;&amp;usbphy 1&gt;;
phy-names = &quot;usb&quot;;
status = &quot;disabled&quot;;
};</code></pre></div>
<p><em>How helpful is all this?</em></p>
<p>Super helpful! The above Device Tree says that the <strong>PinePhone USB Drivers</strong> we seek in U-Boot Bootloader are…</p>
<ul>
<li>
<p><strong>USB PHY</strong> (Physical Layer): “allwinner,sun50i-a64-usb-phy”</p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/phy/allwinner/phy-sun4i-usb.c#L654">phy/allwinner/phy-sun4i-usb.c</a></p>
</li>
<li>
<p><strong>USB EHCI</strong> (Enhanced Host Controller Interface): “allwinner,sun50i-a64-ehci”, “generic-ehci”</p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/usb/host/ehci-generic.c#L160">usb/host/ehci-generic.c</a></p>
</li>
<li>
<p><strong>USB OTG</strong> (On-The-Go): “allwinner,sun8i-a33-musb”</p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/usb/musb-new/sunxi.c#L527">usb/musb-new/sunxi.c</a></p>
<p><a href="https://lupyuen.github.io/articles/usb3#pinephone-usb-controller">(We skip USB OTG)</a></p>
</li>
</ul>
<p>Lets look inside the PinePhone USB Drivers for U-Boot…</p>
<h1 id="power-on-the-usb-controller"><a class="doc-anchor" href="#power-on-the-usb-controller">§</a>8 Power On the USB Controller</h1>
<p><em>Whats inside the PinePhone USB Drivers for U-Boot Bootloader?</em></p>
<p>Earlier we searched for the <a href="https://lupyuen.github.io/articles/usb3#pinephone-usb-drivers-in-u-boot-bootloader"><strong>PinePhone USB Drivers</strong></a> inside U-Boot Bootloader and we found these…</p>
<ul>
<li>
<p>Driver for <strong>USB PHY</strong> (Physical Layer):</p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/phy/allwinner/phy-sun4i-usb.c#L654">phy/allwinner/phy-sun4i-usb.c</a></p>
</li>
<li>
<p>Driver for <strong>USB EHCI</strong> (Enhanced Host Controller Interface):</p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/usb/host/ehci-generic.c#L160">usb/host/ehci-generic.c</a></p>
</li>
</ul>
<p>We skip the USB OTG Driver because were only interested in the <a href="https://lupyuen.github.io/articles/usb3#pinephone-usb-controller"><strong>EHCI Driver (Non-OTG)</strong></a> for PinePhone.</p>
<p><em>USB PHY Driver looks interesting… Its specific to PinePhone?</em></p>
<p>The <strong>USB PHY Driver</strong> handles the <strong>Physical Layer</strong> (electrical wiring) that connects to the USB Controller.</p>
<p>To power on the USB Controller ourselves, lets look inside the <strong>USB PHY Driver</strong>: <a href="https://github.com/u-boot/u-boot/blob/master/drivers/phy/allwinner/phy-sun4i-usb.c#L259-L327">sun4i_usb_phy_init</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Init the USB Physical Layer for PinePhone
static int sun4i_usb_phy_init(struct phy *phy) {
...
// Enable the USB Clocks
clk_enable(&amp;usb_phy-&gt;clocks);
...
// Deassert the USB Resets
reset_deassert(&amp;usb_phy-&gt;resets);</code></pre></div>
<p>In the code above, U-Boot Bootloader will…</p>
<ul>
<li>
<p>Enable the <strong>USB Clocks</strong> for PinePhone</p>
</li>
<li>
<p>Deassert the <strong>USB Resets</strong> for PinePhone</p>
<p>(Deactivate the Reset Signal)</p>
</li>
</ul>
<p>Well come back to these in a while. Then U-Boot does this…</p>
<div class="example-wrap"><pre class="language-c"><code> // Check the Allwinner SoC
if (data-&gt;cfg-&gt;type == sun8i_a83t_phy ||
data-&gt;cfg-&gt;type == sun50i_h6_phy) {
// Skip this part because PinePhone is `sun50i_a64_phy`
...
} else {
// Set PHY_RES45_CAL for Port USB0
if (usb_phy-&gt;id == 0)
sun4i_usb_phy_write(phy, PHY_RES45_CAL_EN,
PHY_RES45_CAL_DATA,
PHY_RES45_CAL_LEN);
// Set USB PHY Magnitude and Rate
sun4i_usb_phy_write(phy, PHY_TX_AMPLITUDE_TUNE,
PHY_TX_MAGNITUDE | PHY_TX_RATE,
PHY_TX_AMPLITUDE_LEN);
// Disconnect USB PHY Threshold Adjustment
sun4i_usb_phy_write(phy, PHY_DISCON_TH_SEL,
data-&gt;cfg-&gt;disc_thresh, PHY_DISCON_TH_LEN);
}</code></pre></div>
<p>Which will…</p>
<ul>
<li>
<p>Set <strong>PHY_RES45_CAL</strong> for Port USB0 (Why?)</p>
</li>
<li>
<p>Set USB PHY <a href="https://github.com/lupyuen/pinephone-nuttx-usb#set-usb-magnitude--rate--threshold"><strong>Magnitude and Rate</strong></a></p>
</li>
<li>
<p>Disconnect USB PHY <strong>Threshold Adjustment</strong> (Why?)</p>
</li>
</ul>
<p>Finally U-Boot does this…</p>
<div class="example-wrap"><pre class="language-c"><code>#ifdef CONFIG_USB_MUSB_SUNXI
// Skip this part because `CONFIG_USB_MUSB_SUNXI` is undefined
...
#else
// Enable USB PHY Bypass
sun4i_usb_phy_passby(phy, true);
// Route PHY0 to HCI to allow USB host
if (data-&gt;cfg-&gt;phy0_dual_route)
sun4i_usb_phy0_reroute(data, false);
#endif
return 0;
}</code></pre></div>
<p>Which will…</p>
<ul>
<li>
<p>Enable <strong>USB PHY Bypass</strong></p>
</li>
<li>
<p>Route <strong>USB PHY0 to EHCI</strong> (instead of Mentor Graphics OTG)</p>
</li>
</ul>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb#usb-controller-configuration">(<strong>phy0_dual_route</strong> is true for PinePhone)</a></p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/phy/allwinner/phy-sun4i-usb.c#L190-L215">(<strong>sun4i_usb_phy_passby</strong> is defined here)</a></p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/phy/allwinner/phy-sun4i-usb.c#L244-L257">(<strong>sun4i_usb_phy0_reroute</strong> is here)</a></p>
<p><em>Whats CONFIG_USB_MUSB_SUNXI?</em></p>
<p><strong>CONFIG_USB_MUSB_SUNXI</strong> enables support for the Mentor Graphics (MUSB) OTG Controller…</p>
<div class="example-wrap"><pre class="language-text"><code>config USB_MUSB_SUNXI
bool &quot;Enable sunxi OTG / DRC USB controller&quot;
depends on ARCH_SUNXI
select USB_MUSB_PIO_ONLY
default y
---help---
Say y here to enable support for the sunxi OTG / DRC USB controller
used on almost all sunxi boards.</code></pre></div>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/usb/musb-new/Kconfig#L68-L75">(Source)</a></p>
<p>We assume <strong>CONFIG_USB_MUSB_SUNXI</strong> is disabled because we wont be using USB OTG for NuttX (yet).</p>
<p>How exactly do we power on the USB Controller, via the USB Clocks and USB Resets? Lets find out…</p>
<h1 id="enable-usb-controller-clocks"><a class="doc-anchor" href="#enable-usb-controller-clocks">§</a>9 Enable USB Controller Clocks</h1>
<p><em>What are the USB Clocks for PinePhone?</em></p>
<p>Earlier we looked at the <a href="https://lupyuen.github.io/articles/usb3#power-on-the-usb-controller"><strong>PinePhone USB PHY Driver for U-Boot</strong></a></p>
<p>And we saw this code that will enable the <strong>USB Clocks</strong>: <a href="https://github.com/u-boot/u-boot/blob/master/drivers/phy/allwinner/phy-sun4i-usb.c#L266-L271">sun4i_usb_phy_init</a></p>
<div class="example-wrap"><pre class="language-c"><code>clk_enable(&amp;usb_phy-&gt;clocks);</code></pre></div>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/clk/sunxi/clk_sunxi.c#L58-L61">(<strong>clk_enable</strong> is defined here)</a></p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/clk/sunxi/clk_sunxi.c#L30-L56">(Which calls <strong>sunxi_set_gate</strong>)</a></p>
<p><em>Whats usb_phy→clocks?</em></p>
<p>According to the <a href="https://lupyuen.github.io/articles/usb3#pinephone-usb-drivers-in-u-boot-bootloader"><strong>PinePhone Device Tree</strong></a>, the USB Clocks are…</p>
<ul>
<li>
<p><strong>usb0_phy:</strong> CLK_USB_PHY0</p>
</li>
<li>
<p><strong>usb1_phy:</strong> CLK_USB_PHY1</p>
</li>
<li>
<p><strong>EHCI0:</strong> CLK_BUS_OHCI0, CLK_BUS_EHCI0, CLK_USB_OHCI0</p>
</li>
<li>
<p><strong>EHCI1:</strong> CLK_BUS_OHCI1, CLK_BUS_EHCI1, CLK_USB_OHCI1</p>
</li>
</ul>
<p>These are the <strong>USB Clocks</strong> that our NuttX EHCI Driver should enable.</p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb#enable-usb-controller-clocks">(More about this)</a></p>
<p><em>What clickers are these: CLK_USB and CLK_BUS?</em></p>
<p>They refer to the <strong>Clock Control Unit (CCU) Registers</strong> defined in the <a href="https://github.com/lupyuen/pinephone-nuttx/releases/download/doc/Allwinner_A64_User_Manual_V1.1.pdf"><strong>Allwinner A64 User Manual</strong></a>. (Page 81)</p>
<p>CCU Base Address is <strong><code>0x01C2</code> <code>0000</code></strong></p>
<p><em>What are the addresses of these CCU Registers?</em></p>
<p>U-Boot tells us the <strong>addresses of the CCU Registers</strong> for USB Clocks: <a href="https://github.com/u-boot/u-boot/blob/master/drivers/clk/sunxi/clk_a64.c#L16-L66">clk_a64.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// USB Clocks: CCU Offset and Bit Number
static const struct ccu_clk_gate a64_gates[] = {
[CLK_BUS_EHCI0] = GATE(0x060, BIT(24)),
[CLK_BUS_EHCI1] = GATE(0x060, BIT(25)),
[CLK_BUS_OHCI0] = GATE(0x060, BIT(28)),
[CLK_BUS_OHCI1] = GATE(0x060, BIT(29)),
[CLK_USB_PHY0] = GATE(0x0cc, BIT(8)),
[CLK_USB_PHY1] = GATE(0x0cc, BIT(9)),
[CLK_USB_OHCI0] = GATE(0x0cc, BIT(16)),
[CLK_USB_OHCI1] = GATE(0x0cc, BIT(17)),</code></pre></div>
<p>So to enable the USB Clock <strong>CLK_BUS_EHCI0</strong>, well set <strong>Bit 24</strong> of the CCU Register at <strong><code>0x060</code> + <code>0x01C2</code> <code>0000</code></strong>.</p>
<p>These CCU Registers are also mentioned in the <a href="https://github.com/lupyuen/pinephone-nuttx/releases/download/doc/Allwinner_A64_User_Manual_V1.1.pdf"><strong>Allwinner A64 User Manual</strong></a>, buried deep inside Pages 81 to 147.</p>
<p><em>How will NuttX enable the USB Clocks?</em></p>
<p>Our <strong>NuttX EHCI Driver</strong> will enable the USB Clocks like this: <a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/0e1632ed351975a6432b7e4fde1857d6bcc0940a/a64_usbhost.c#L138-L193">a64_usbhost.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Allwinner A64 Clock Control Unit (CCU)
#define A64_CCU_ADDR 0x01c20000
// Enable the USB Clocks for PinePhone
static void a64_usbhost_clk_enable(void) {
// Enable usb0_phy: CLK_USB_PHY0
// 0x0cc BIT(8)
#define CLK_USB_PHY0 (A64_CCU_ADDR + 0x0cc)
#define CLK_USB_PHY0_BIT 8
set_bit(CLK_USB_PHY0, CLK_USB_PHY0_BIT);
// Enable EHCI0: CLK_BUS_OHCI0
// 0x060 BIT(28)
#define CLK_BUS_OHCI0 (A64_CCU_ADDR + 0x060)
#define CLK_BUS_OHCI0_BIT 28
set_bit(CLK_BUS_OHCI0, CLK_BUS_OHCI0_BIT);
// Omitted: Do the same for...
// CLK_USB_PHY0, CLK_USB_PHY1
// CLK_BUS_OHCI0, CLK_BUS_EHCI0, CLK_USB_OHCI0
// CLK_BUS_OHCI1, CLK_BUS_EHCI1, CLK_USB_OHCI1
// Yeah this looks excessive. We probably need only
// USB PHY1, EHCI1 and OHCI1.</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/0e1632ed351975a6432b7e4fde1857d6bcc0940a/a64_usbhost.c#L131-L136">(<strong>set_bit(addr, bit)</strong> sets the bit at an address)</a></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/0e1632ed351975a6432b7e4fde1857d6bcc0940a/a64_usbhost.c#L261-L279">(<strong>a64_usbhost_clk_enable</strong> is called by <strong>a64_usbhost_initialize</strong>)</a></p>
<p>Now we do the same for the USB Resets…</p>
<p>TODO: What about OHCI1_12M_SRC_SEL and OHCI0_12M_SRC_SEL? (Allwinner A64 User Manual, Page 113)</p>
<h1 id="reset-usb-controller"><a class="doc-anchor" href="#reset-usb-controller">§</a>10 Reset USB Controller</h1>
<p><em>What are the USB Resets for PinePhone?</em></p>
<p>A while ago we looked at the <a href="https://lupyuen.github.io/articles/usb3#power-on-the-usb-controller"><strong>PinePhone USB PHY Driver for U-Boot</strong></a></p>
<p>And we saw this code that will deassert (deactivate) the <strong>USB Resets</strong>: <a href="https://github.com/u-boot/u-boot/blob/master/drivers/phy/allwinner/phy-sun4i-usb.c#L273-L278">sun4i_usb_phy_init</a></p>
<div class="example-wrap"><pre class="language-c"><code>reset_deassert(&amp;usb_phy-&gt;resets);</code></pre></div>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/reset/reset-uclass.c#L207-L214">(<strong>reset_deassert</strong> is defined here)</a></p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/reset/reset-sunxi.c#L71-L75">(Which calls <strong>rst_deassert</strong>)</a></p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/reset/reset-sunxi.c#L66-L69">(Which calls <strong>sunxi_reset_deassert</strong>)</a></p>
<p><a href="https://github.com/u-boot/u-boot/blob/master/drivers/reset/reset-sunxi.c#L36-L59">(Which calls <strong>sunxi_set_reset</strong> phew!)</a></p>
<p><em>Whats usb_phy→resets?</em></p>
<p>According to the <a href="https://lupyuen.github.io/articles/usb3#pinephone-usb-drivers-in-u-boot-bootloader"><strong>PinePhone Device Tree</strong></a>, the USB Resets are…</p>
<ul>
<li>
<p><strong>usb0_reset:</strong> RST_USB_PHY0</p>
</li>
<li>
<p><strong>usb1_reset:</strong> RST_USB_PHY1</p>
</li>
<li>
<p><strong>EHCI0:</strong> RST_BUS_OHCI0, RST_BUS_EHCI0</p>
</li>
<li>
<p><strong>EHCI1:</strong> RST_BUS_OHCI1, RST_BUS_EHCI1</p>
</li>
</ul>
<p>These are the <strong>USB Resets</strong> that our NuttX EHCI Driver shall deassert.</p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb#reset-usb-controller">(More about this)</a></p>
<p><em>What exactly are RST_USB and RST_BUS?</em></p>
<p>Theyre the <strong>Clock Control Unit (CCU) Registers</strong> defined in the <a href="https://github.com/lupyuen/pinephone-nuttx/releases/download/doc/Allwinner_A64_User_Manual_V1.1.pdf"><strong>Allwinner A64 User Manual</strong></a>. (Page 81)</p>
<p>CCU Base Address (once again) is <strong><code>0x01C2</code> <code>0000</code></strong></p>
<p><em>What are the addresses of these CCU Registers?</em></p>
<p>U-Boot helpfully reveals the <strong>addresses of the CCU Registers</strong> for USB Resets: <a href="https://github.com/u-boot/u-boot/blob/master/drivers/clk/sunxi/clk_a64.c#L68-L100">clk_a64.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// USB Resets: CCU Offset and Bit Number
static const struct ccu_reset a64_resets[] = {
[RST_USB_PHY0] = RESET(0x0cc, BIT(0)),
[RST_USB_PHY1] = RESET(0x0cc, BIT(1)),
[RST_BUS_EHCI0] = RESET(0x2c0, BIT(24)),
[RST_BUS_EHCI1] = RESET(0x2c0, BIT(25)),
[RST_BUS_OHCI0] = RESET(0x2c0, BIT(28)),
[RST_BUS_OHCI1] = RESET(0x2c0, BIT(29)),</code></pre></div>
<p>Hence to deassert the USB Reset <strong>RST_USB_PHY0</strong>, well set <strong>Bit 0</strong> of the CCU Register at <strong><code>0x0CC</code> + <code>0x01C2</code> <code>0000</code></strong>.</p>
<p>These CCU Registers are also mentioned in the <a href="https://github.com/lupyuen/pinephone-nuttx/releases/download/doc/Allwinner_A64_User_Manual_V1.1.pdf"><strong>Allwinner A64 User Manual</strong></a>, buried deep inside Pages 81 to 147.</p>
<p><em>How will NuttX deassert the USB Resets?</em></p>
<p>Our <strong>NuttX EHCI Driver</strong> will deassert the USB Resets like so: <a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/0e1632ed351975a6432b7e4fde1857d6bcc0940a/a64_usbhost.c#L206-L249">a64_usbhost.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Allwinner A64 Clock Control Unit (CCU)
#define A64_CCU_ADDR 0x01c20000
// Deassert the USB Resets for PinePhone
static void a64_usbhost_reset_deassert(void) {
// Deassert usb0_reset: RST_USB_PHY0
// 0x0cc BIT(0)
#define RST_USB_PHY0 (A64_CCU_ADDR + 0x0cc)
#define RST_USB_PHY0_BIT 0
set_bit(RST_USB_PHY0, RST_USB_PHY0_BIT);
// Deassert EHCI0: RST_BUS_OHCI0
// 0x2c0 BIT(28)
#define RST_BUS_OHCI0 (A64_CCU_ADDR + 0x2c0)
#define RST_BUS_OHCI0_BIT 28
set_bit(RST_BUS_OHCI0, RST_BUS_OHCI0_BIT);
// Omitted: Do the same for...
// RST_USB_PHY0, RST_USB_PHY1
// RST_BUS_OHCI0, RST_BUS_EHCI0
// RST_BUS_OHCI1, RST_BUS_EHCI1
// Yeah this looks excessive. We probably need only
// USB PHY1, EHCI1 and OHCI1.</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/0e1632ed351975a6432b7e4fde1857d6bcc0940a/a64_usbhost.c#L131-L136">(<strong>set_bit(addr, bit)</strong> sets the bit at an address)</a></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/0e1632ed351975a6432b7e4fde1857d6bcc0940a/a64_usbhost.c#L261-L279">(<strong>a64_usbhost_clk_enable</strong> is called by <strong>a64_usbhost_initialize</strong>)</a></p>
<p>Weve powered up the USB Controller via the USB Clocks and USB Resets. Lets test this!</p>
<p><img src="https://lupyuen.github.io/images/usb3-run.png" alt="Booting NuttX EHCI Driver on PinePhone" /></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/5238bc5246bcae896883f056d24691ebaa050f83/README.md#output-log"><em>Booting NuttX EHCI Driver on PinePhone</em></a></p>
<h1 id="nuttx-ehci-driver-starts-ok-on-pinephone"><a class="doc-anchor" href="#nuttx-ehci-driver-starts-ok-on-pinephone">§</a>11 NuttX EHCI Driver Starts OK on PinePhone</h1>
<p><em>Now that weve powered up the USB Controller on PinePhone…</em></p>
<p><em>Will the EHCI Driver start correctly on NuttX?</em></p>
<p>Remember the <strong>NuttX EHCI Driver</strong> failed during PinePhone startup…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/usb3#halt-timeout-for-usb-controller"><strong>“Halt Timeout for USB Controller”</strong></a></li>
</ul>
<p>Then we discovered how the <strong>U-Boot Bootloader</strong> enables the <strong>USB Clocks</strong> and deasserts the <strong>USB Resets</strong></p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/usb3#enable-usb-controller-clocks"><strong>“Enable USB Controller Clocks”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/articles/usb3#reset-usb-controller"><strong>“Reset USB Controller”</strong></a></p>
</li>
</ul>
<p>So we did the same for <strong>NuttX on PinePhone</strong>: <a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/0e1632ed351975a6432b7e4fde1857d6bcc0940a/a64_usbhost.c#L261-L279">a64_usbhost.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Init the USB EHCI Host at NuttX Startup
int a64_usbhost_initialize(void) {
// Enable the USB Clocks for PinePhone
a64_usbhost_clk_enable();
// Deassert the USB Resets for PinePhone
a64_usbhost_reset_deassert();</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/0e1632ed351975a6432b7e4fde1857d6bcc0940a/a64_usbhost.c#L138-L193">(<strong>a64_usbhost_clk_enable</strong> is defined here)</a></p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/0e1632ed351975a6432b7e4fde1857d6bcc0940a/a64_usbhost.c#L206-L249">(<strong>a64_usbhost_reset_deassert</strong> is defined here)</a></p>
<p>And now the NuttX EHCI Driver <strong>starts OK on PinePhone</strong> yay! 🎉</p>
<p>Heres the log…</p>
<div class="example-wrap"><pre class="language-text"><code>a64_usbhost_clk_enable:
CLK_USB_PHY0, CLK_USB_PHY1
CLK_BUS_OHCI0, CLK_BUS_EHCI0
CLK_USB_OHCI0, CLK_BUS_OHCI1
CLK_BUS_EHCI1, CLK_USB_OHCI1
a64_usbhost_reset_deassert:
RST_USB_PHY0, RST_USB_PHY1
RST_BUS_OHCI0, RST_BUS_EHCI0
RST_BUS_OHCI1, RST_BUS_EHCI1</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/5238bc5246bcae896883f056d24691ebaa050f83/README.md#output-log">(See the Complete Log)</a></p>
<p>The log above shows NuttX enabling the <strong>USB Clocks</strong> and deasserting the <strong>USB Resets</strong> for…</p>
<ul>
<li>
<p>USB PHY0 and USB PHY1</p>
</li>
<li>
<p>EHCI0 and OHCI0</p>
</li>
<li>
<p>EHCI1 and OHCI1</p>
</li>
</ul>
<p>(Yeah this looks excessive. We probably need only USB PHY1, EHCI1 and OHCI1)</p>
<p>Then the <strong>NuttX EHCI Driver</strong> starts…</p>
<div class="example-wrap"><pre class="language-text"><code>usbhost_registerclass:
Registering class:0x40124838 nids:2
EHCI Initializing EHCI Stack
EHCI HCIVERSION 1.00
EHCI nports=1, HCSPARAMS=1101
EHCI HCCPARAMS=00a026
EHCI USB EHCI Initialized
NuttShell (NSH) NuttX-12.0.3
nsh&gt; </code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/5238bc5246bcae896883f056d24691ebaa050f83/README.md#output-log">(See the Complete Log)</a></p>
<p>Which says that NuttX has <strong>successfully started the EHCI Controller</strong>. Yay!</p>
<p><em>But does the driver actually work?</em></p>
<p>Well find out soon as we <strong>test the NuttX EHCI Driver</strong> on PinePhone! Our test plan…</p>
<ul>
<li>
<p><strong>Enumerate the USB Devices</strong> on PinePhone</p>
<p><a href="https://lupyuen.github.io/articles/usb3#pinephone-usb-controller">(Especially the LTE Modem)</a></p>
</li>
<li>
<p><strong>Handle the USB Interrupts</strong> on PinePhone</p>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/main/a64_ehci.c#L5325-L5345">(See this)</a></p>
</li>
<li>
<p>Verify the values of <strong>HCSPARAMS</strong> and <strong>HCCPARAMS</strong></p>
<div class="example-wrap"><pre class="language-text"><code>EHCI nports=1, HCSPARAMS=1101
EHCI HCCPARAMS=00a026</code></pre></div>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb/blob/5238bc5246bcae896883f056d24691ebaa050f83/README.md#output-log">(Based on the log)</a></p>
</li>
</ul>
<p>The USB Descriptors for PinePhones LTE Modem are defined here…</p>
<ul>
<li><a href="https://github.com/lupyuen/lupyuen.github.io/blob/master/images/Quectel_EC2x&amp;EG2x-G&amp;EG9x_Series_USB_Descriptor_Introduction_V1.1.pdf"><strong>EG25-G USB Descriptors</strong></a></li>
</ul>
<p>Check out the progress here…</p>
<ul>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb#decode-ehci-register-values"><strong>“Decode EHCI Register Values”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb#enumerate-usb-devices-on-pinephone"><strong>“Enumerate USB Devices on PinePhone”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb#handle-usb-interrupt"><strong>“Handle USB Interrupt”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb#power-on-lte-modem"><strong>“Power On LTE Modem”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx-usb#testing-cdc-acm"><strong>“Testing CDC ACM”</strong></a></p>
</li>
</ul>
<h1 id="whats-next"><a class="doc-anchor" href="#whats-next">§</a>12 Whats Next</h1>
<p>(I promised to reward myself with a <a href="https://qoto.org/@lupyuen/110109772143549589"><strong>Bread Machine</strong></a> when the NuttX EHCI Driver boots OK on PinePhone… Time to go shopping! 😀)</p>
<p>Today we made a significant breakthrough in supporting <strong>PinePhone USB on NuttX</strong></p>
<ul>
<li>
<p>NuttX USB Driver now <a href="https://lupyuen.github.io/articles/usb3#nuttx-ehci-driver-starts-ok-on-pinephone"><strong>boots OK on PinePhone!</strong></a> 🎉</p>
</li>
<li>
<p>We tweaked slightly the NuttX Driver for <a href="https://lupyuen.github.io/articles/usb3#ehci-driver-from-apache-nuttx"><strong>USB Enhanced Host Controller Interface</strong></a> (EHCI)</p>
</li>
<li>
<p>Which is a lot simpler than <a href="https://lupyuen.github.io/articles/usb3#ehci-is-simpler-than-usb-on-the-go"><strong>USB On-The-Go</strong></a> (OTG)</p>
</li>
<li>
<p>Remember to enable the <a href="https://lupyuen.github.io/articles/usb3#enable-usb-controller-clocks"><strong>USB Clocks</strong></a></p>
</li>
<li>
<p>And deassert the <a href="https://lupyuen.github.io/articles/usb3#reset-usb-controller"><strong>USB Resets</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/articles/usb3#pinephone-usb-drivers-in-u-boot-bootloader"><strong>U-Boot Bootloader</strong></a> is a terrific resource for PinePhone USB</p>
</li>
<li>
<p>Were one step closer to our dream of a <a href="https://lupyuen.github.io/articles/usb2#pinephone--nuttx--feature-phone"><strong>NuttX Feature Phone</strong></a>!</p>
</li>
</ul>
<p>Meanwhile please check out the other articles on NuttX for PinePhone…</p>
<ul>
<li><a href="https://github.com/lupyuen/pinephone-nuttx"><strong>“Apache NuttX RTOS for PinePhone”</strong></a></li>
</ul>
<p>Many Thanks to my <a href="https://lupyuen.github.io/articles/sponsor"><strong>GitHub Sponsors</strong></a> for supporting my work! This article wouldnt have been possible without your support.</p>
<p>Special Thanks to <a href="https://news.apache.org/foundation/entry/the-apache-software-foundation-announced-apache-nuttx12-0"><strong>TL Lim</strong></a> for the inspiring and invigorating chat! 🙂</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/sponsor"><strong>Sponsor me a coffee</strong></a></p>
</li>
<li>
<p><a href="https://www.reddit.com/r/PINE64official/comments/11zn3zx/nuttx_rtos_for_pinephone_simpler_usb_with_ehci/"><strong>Discuss this article on Reddit</strong></a></p>
</li>
<li>
<p><a href="https://news.ycombinator.com/item?id=35275904"><strong>Discuss this article on Hacker News</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-ox64"><strong>My Current Project: “Apache NuttX RTOS for Ox64 BL808”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-star64"><strong>My Other Project: “NuttX for Star64 JH7110”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx"><strong>Older Project: “NuttX for PinePhone”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/articles/sourdough"><strong>My Sourdough Recipe</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io"><strong>Check out my articles</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/rss.xml"><strong>RSS Feed</strong></a></p>
</li>
</ul>
<p><em>Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…</em></p>
<p><a href="https://github.com/lupyuen/lupyuen.github.io/blob/master/src/usb3.md"><strong>lupyuen.github.io/src/usb3.md</strong></a></p>
<!-- Begin scripts/rustdoc-after.html: Post-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker and Prism Theme -->
<script src="../theme.js"></script>
<script src="../prism.js"></script>
<!-- Theme Picker and Prism Theme -->
<!-- End scripts/rustdoc-after.html -->
</body>
</html>