Doc: migration filesystem doc

Migrate these pages from Confluence wiki to official wiki:

* https://cwiki.apache.org/confluence/display/NUTTX/Special+Files+and+Device+Numbers
* https://cwiki.apache.org/confluence/display/NUTTX/Pseudo+File+System
* https://cwiki.apache.org/confluence/display/NUTTX/How+NXFFS+Works
* https://cwiki.apache.org/confluence/display/NUTTX/NuttX+File+System
* https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=139630111

Documentation/components/filesystem/how_nxffs_works.rst

@@ -0,0 +1,11 @@
+===============
+How NXFFS works
+===============
+
+.. warning:: 
+    Migrated from: 
+    https://cwiki.apache.org/confluence/display/NUTTX/How+NXFFS+Works
+
+.. image:: nxffs/nxffs_page_0.png
+.. image:: nxffs/nxffs_page_1.png
+.. image:: nxffs/nxffs_page_2.png

Documentation/components/filesystem/index.rst

@@ -553,6 +553,10 @@ NuttX provides support for a variety of file systems out of the box.
   userfs.rst
   zipfs.rst
   inotify.rst
+  nuttxfs.rst
+  nxflat.rst
+  pseudofs.rst
+  special_files_dev_num.rst
 
 FS Categories
 -------------

Documentation/components/filesystem/nuttxfs.rst

@@ -0,0 +1,11 @@
+========
+Nuttx FS
+========
+
+.. warning:: 
+    Migrated from: 
+    https://cwiki.apache.org/confluence/display/NUTTX/NuttX+File+System
+
+.. image:: nuttxfs/nuttxfs_page_0.png
+.. image:: nuttxfs/nuttxfs_page_1.png
+.. image:: nuttxfs/nuttxfs_page_2.png

Documentation/components/filesystem/nxffs.rst

@@ -184,3 +184,10 @@ Things to Do
   implementation of a proper truncate() method which should alter the
   size of a previously written file.  There is some fragmentary logic in
   place but even this is conditioned out with __NO_TRUNCATE_SUPPORT__.
+
+
+Further reference
+=================
+
+.. toctree::
+  how_nxffs_works.rst

Documentation/components/filesystem/nxflat.rst

@@ -0,0 +1,180 @@
+=================================
+NuttX FLAT Binary Format (NXFLAT)
+=================================
+
+.. warning:: 
+    Migrated from: 
+    https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=139630111
+
+Overview
+========
+
+NuttX supports a configurable :doc:`binary loader <../binfmt>` .
+This binary loader supports loading and executing binary objects from 
+the file system.
+The NuttX binary loader is capable of supporting multiple binary formats.
+One of of those binary formats is NXFLAT, the top of this Wiki page.
+
+NXFLAT is a customized and simplified version of binary format 
+implemented a few years ago called XFLAT.
+With the NXFLAT binary format you will be able to do the following:
+
+* Place separately linked programs in a file system, and
+
+* Execute those programs by dynamically linking them to the 
+  base NuttX code.
+
+This allows you to extend the NuttX base code after it has been 
+written into FLASH.
+One motivation for implementing NXFLAT is support clean CGI under 
+an HTTPD server.
+
+This feature is especially attractive when combined with the 
+NuttX ROMFS support:
+ROMFS allows you to execute programs in place (XIP) in flash 
+without copying anything other than the .data section to RAM.
+In fact, the initial NXFLAT release only worked on ROMFS.
+Later extensions also support execution NXFLAT binaries from 
+an SRAM copy as well.
+
+This NuttX feature includes:
+
+* A dynamic loader that is built into the NuttX core (See SVN).
+* Minor changes to RTOS to support position independent code, and
+* A linker to bind ELF binaries to produce the 
+  NXFLAT binary format (See SVN).
+
+Toolchain Compatibility Problem
+===============================
+
+Description
+-----------
+
+NXFLAT flat requires a specific kind of position independence. 
+The ARM family of GCC toolchains has historically supported 
+this method of position independence: All code addresses are 
+accessed relative to the Program Counter (PC) and a special, 
+`PIC register` (usually ``r10``) is used to access all data. To 
+load or store a data value, the contents of ``r10``, the PIC 
+base, is added to a constant, position-independent offset 
+to produce the absolute address of the data.
+
+.. image:: nxflat/nxflat-addressing.png
+
+The `Global Offset Table` (GOT) is a special data structure that 
+resides in D-Space. So PIC-base relative addressing may also 
+be specified as GOT-Relative addressing (or ``GOTOFF``). The older 
+GCC 4.3.3 GCC compiler, for example, generates ``GOTOFF`` 
+relocations to the constant strings, like:
+
+.. code-block:: asm
+
+    .L3:
+            .word   .LC0(GOTOFF)
+            .word   .LC1(GOTOFF)
+            .word   .LC2(GOTOFF)
+            .word   .LC3(GOTOFF)
+            .word   .LC4(GOTOFF)
+
+Where ``.LC0``, ``.LC1``, ``.LC2``, ``.LC3``, and ``.LC4`` are the labels 
+corresponding to strings in the ``.rodata.str1.1`` section. One 
+consequence of this is that ``.rodata`` must reside in D-Space 
+since it will addressed relative to the GOT (see the section 
+entitled "Read-Only Data in RAM" here).
+
+The newer 4.6.3 GCC compiler, however, generated PC relative 
+relocations to these same strings:
+
+.. code-block::
+
+    .L2:
+        .word   .LC0-(.LPIC0+4)
+        .word   .LC1-(.LPIC1+4)
+        .word   .LC2-(.LPIC2+4)
+        .word   .LC3-(.LPIC4+4)
+        .word   .LC4-(.LPIC5+4)
+
+These are `PC-Relative` relocations. This means that the string 
+data is address not by an offset relative to the PIC register 
+(``r10``) but to the program count (PC). This is good and bad. This 
+is good because it means that ``.rodata.str1.1`` must now can now 
+reside in FLASH with ``.text`` and can be accessed using PC-relative 
+addressing. That can be accomplished by simply moving the ``.rodata`` 
+from the ``.data`` section to the ``.text`` section in the linker script. 
+The NXFLAT linker script is located at 
+``nuttx/binfmt/libnxflat/gnu-nxflat-?.ld``. **NOTE**: There are two 
+linker scripts located at ``nuttx/binfmt/libnxflat/``:
+
+1. ``binfmt/libnxflat/gnu-nxflat-gotoff.ld.`` Older versions 
+   of GCC (at least up to GCC 4.3.3), use GOT-relative 
+   addressing to access RO data. In that case, read-only 
+   data (``.rodata``) must reside in D-Space and this linker 
+   script should be used.
+2. ``binfmt/libnxflat/gnu-nxflat-pcrel.ld.`` Newer versions 
+   of GCC (at least as of GCC 4.6.3), use PC-relative 
+   addressing to access RO data. In that case, read-only 
+   data (``.rodata``) must reside in I-Space and this linker 
+   script should be used.
+
+But this is very bad because a lot of NXFLAT is now broken. 
+For it appears that not just constant strings, but that all 
+data may now be referenced using PC-relative addressing to 
+.bss and .data values. I do not yet know the extent of this 
+problem or the direction that GCC is going but certainly 
+version 4.6.3 cannot be used with NXFLAT.
+
+The workaround for now is to use the older, 4.3.3 OABI 
+compiler. In the long run, this might spell the end 
+to NXFLAT.
+
+Update: Restored GCC Support
+----------------------------
+
+This post was pointed out by Michael Jung:
+
+.. code-block:: bash
+
+    MCU: STM32F4 (ARM Cortex M4)
+    Build environment: arm-none-eabi-gcc 4.8.4 20140725
+    
+    My goal is to build an image that can be run from any properly-aligned
+    offset in internal flash (i.e., position-independent).  I found the
+    following set of gcc flags that achieves this goal:
+    
+        # Generate position independent code.
+        -fPIC
+        
+        # Access bss via the GOT.
+        -mno-pic-data-is-text-relative
+        
+        # GOT is not PC-relative; store GOT location in a register.
+        -msingle-pic-base
+        
+        # Store GOT location in r9.
+        -mpic-register=r9
+
+Reference: https://gcc.gnu.org/ml/gcc-help/2015-07/msg00027.html
+
+Michael has verified that ``-mno-pic-data-is-text-relative`` is, 
+indeed, a solution to the above NXFLAT problem in newer 
+compilers. You simply need to modify the board Make.defs 
+file like:
+
+1. ARCHPICFLAGS = -fpic -msingle-pic-base -mpic-register=r10
+
+.. code-block:: bash
+
+    +ARCHPICFLAGS = -fpic -msingle-pic-base -mpic-register=r10 -mno-pic-data-is-text-relative
+
+NOTE the minor difference from the post: NuttX uses ``r10`` as 
+the PIC base register by default in all configurations.
+
+See this `thread <https://groups.google.com/forum/>`_ for additional information.
+
+References
+----------
+
+* :doc:`NXFLAT <../nxflat>`
+* `XFLATFLAT <http://xflat.sourceforge.net/>`_
+* `FLAT <http://retired.beyondlogic.org/uClinux/bflt.htm>`_
+* `ROMFS <http://romfs.sourceforge.net/>`_

Documentation/components/filesystem/pseudofs.rst

@@ -0,0 +1,211 @@
+==================
+Pseudo File System
+==================
+
+.. warning:: 
+    Migrated from: 
+    https://cwiki.apache.org/confluence/display/NUTTX/Pseudo+File+System
+
+Overview
+========
+
+Pseudo Root File System
+-----------------------
+
+NuttX includes an optional, scalable file system. As a minimum, this may 
+be a simple in-memory, pseudo file system. This is an in-memory file 
+system because it does not require any storage medium or block driver 
+support. Rather, file system contents are generated on-the-fly as 
+referenced via standard file system operations (open, close, read, 
+write, etc.). In this sense, the file system is a pseudo file system 
+(in the same sense that the Linux ``/proc`` file system is also referred 
+to as a pseudo file system).
+
+Any user supplied data or logic can be accessed via the pseudo-file 
+system. Built in support is provided for character, block, and MTD 
+(Memory Technology Device) drivers in the ``/dev`` pseudo file system 
+directory.
+
+Special Files
+-------------
+
+NuttX does not support special files in the way that, say, 
+Linux does. In fact, it is more correct to say that NuttX 
+file systems do not support special files at all.
+
+NuttX does, however, support Linux-like special `device node`, 
+character driver, and block driver files (as well as NuttX-specific 
+mountpoint, named semaphore, message queue, and shared memory 
+special files). However, these are not special files in sense 
+that the term special files is used in a POSIX environment: In 
+NuttX these special files may `only` be created in the root 
+pseudo-file system. For the case of device nodes, see `Device 
+Nodes <https://cwiki.apache.org/confluence/display/NUTTX/Device+Nodes>`_ 
+for further information.
+
+In NuttX, the underlying principle is that all `named resources` 
+appear as special files in the root pseudo-file system and are 
+managed by the VFS.
+
+Mounted Volumes
+---------------
+
+The simple in-memory file system can be extended by mounting block 
+devices that provide access to true file systems backed up via 
+some mass storage device. NuttX supports the standard mount() 
+command that allows a block driver to be bound to a mount point 
+within the pseudo file system and to a file system. At present, 
+NuttX supports the standard VFAT and ROMFS file systems, a 
+special, wear-levelling NuttX FLASH File System (NXFFS), as well 
+as a Network File System client (NFS version 3, UDP).
+
+Comparison to Linux
+-------------------
+
+From a programming perspective, the NuttX file system appears 
+very similar to a Linux file system. However, there is a 
+fundamental difference: The NuttX root file system is a pseudo 
+file system and true file systems may be mounted in the pseudo 
+file system. In the typical Linux installation by comparison, 
+the Linux root file system is a true file system and pseudo 
+file systems may be mounted in the true, root file system. 
+The approach selected by NuttX is intended to support greater 
+scalability from the very tiny platform to the moderate platform.
+
+FAQ
+===
+
+**Question**: I'm wondering why I can't create a directory. If 
+I try to create a dir.
+
+.. code-block:: bash
+
+    mkdir /mnt
+
+I get this,
+
+.. code-block:: bash
+
+    nsh: mkdir: mkdir failed: 2
+
+although if I do this it creates both directories, mnt and sda
+
+.. code-block:: bash
+
+    mount -t vfat /dev/mmcsd0 /mnt/sda
+
+**Answer**: This is because the top level directories are part of a 
+`pseudo-filesystem` – like the Linux ``proc/`` or ``sys/`` file systems. 
+But the NuttX pseudo-file system begins at the top level ``/``.
+
+What that really means is that you do must have 
+``CONFIG_DISABLE_PSEUDOFS_OPERATIONS`` selected. Because you 
+can normally create directories in the pseudo-filesystem 
+with not problem:
+
+.. code-block:: bash
+
+    NuttShell (NSH) NuttX-9.0.0
+    nsh> mkdir /mnt
+    nsh> ls
+    /:
+    dev/
+    etc/
+    mnt/
+    proc/
+    tmp/
+    nsh> ls mnt
+    /mnt:
+    nsh>
+
+But lets assume that you do have operations on the pseudo-file 
+system disabled. Why doesn't it work? There is no `real` media 
+there so you cannot create a file there or create any directories 
+there. The ``mount`` command is special, it knows how to create mount 
+points in the pseudo-file system.
+
+The pseudo-file system is just a tree structure in RAM. 
+It serves two purposes: (1) you don't have to have a real 
+file system to use NuttX.
+It comes up out-of-the-box with usable (but limited) 
+pseudo-file system. That allows a little more civilized 
+programming environment on even very resource limited MCUs. 
+And (2) this pseudo-file system is a place where all special 
+NuttX files are retained: Character drivers, block drivers, 
+and mount points.
+
+The NuttX top-level pseudo-filesystem creates the `illusion` of 
+directories and provides a consistent, seamless semantic for 
+interacting with mounted file systems. If there is a file 
+called ``hello.txt`` in your volume mounted at ``/mnt/sda``, then:
+
+``/mnt`` - is a `node` in the pseudo-filesystem that does 
+nothing but contain the name mnt and provide links 
+to things `under` ``mnt``.
+
+``/mnt/sda`` - This refers to a node that contains the name 
+sda that can be found `under` the node with the name mnt.
+This node is a special `mountpoint node` in the pseudo-filesystem.
+It contains the methods needed to interact will real file system.
+Everything `below` ``/mnt/sda`` is in the physical media.
+
+``/mnt/sda/hello.txt`` - This, then refers to the file 
+``hello.txt`` at the relative path ``hello.txt`` on the mounted media.
+The transition from the pseudo-filesystem to the 
+real media is seamless.
+
+This is a little different from Linux: Linux always 
+has to boot up with a `real` file system – even if it 
+is only a initrd RAM disk.
+In Linux, these special files (links, drivers, pipes, 
+etc.) reside on real media and can reside in any 
+Linux-compatible filesystem.
+
+Normal ``mkdir`` can only work if there is a `real` filesystem 
+at the location. There are no real directories in the 
+pseudo-filesystem. The pseudo-filesystem does support 
+`nodes` that look like directories and have some of the 
+properties of directories (like the node ``/mnt`` mentioned 
+above). But this is really an illusion.
+
+If ``CONFIG_DISABLE_PSEUDOFS_OPERATIONS`` is not enabled, 
+then NuttX adds the capability to create new, empty `nodes` 
+in the pseudo-filesystem using ``mkdir``, completing the illusion.
+
+[On the other hand, all directories are really an 
+`illusion` in a way and I suppose that in that sense 
+these nodes the pseudo-filesystem are just as `real` 
+as any other directory.]
+
+After you mount the SD card at ``/mnt/sda``, then you can do:
+
+.. code-block:: bash
+
+    mkdir /mnt/sda/newdir
+
+That should work fine and should create a directory at 
+the relative path ``newdir`` in the mounted volume.
+
+There are a few other special NSH commands like mount that 
+can change the pseudo-filesystem. Like ``losetup``, ``mkfifo``, 
+``mkrd``, ``umount``, etc.
+In fact, these commands `only` work in the pseudo-filesystem. 
+Try them in ``/mnt/sda``... they won't work.
+
+But none of the `normal` commands that modify files or directories 
+will work in the pseudo-filesystem: ``mkdir``, ``mv``, ``rm``, ``rmdir``. 
+These all require real media. They will not work in the 
+pseudo-filesystem, but will work in ``/mnt/sda``.
+
+And trying to pipe to something in the pseudo-filesystem 
+will also fail. You cannot do this, for example:
+
+.. code-block:: bash
+
+    NuttShell (NSH) NuttX-6.20
+    nsh> cat "Hello, World!" >/hello.text
+    nsh: cat: open failed: 22
+    nsh>
+
+See also NxFileSystem in 
+`Porting Guide <https://cwiki.apache.org/confluence/display/NUTTX/Porting+Guide>`_

Documentation/components/filesystem/special_files_dev_num.rst

@@ -0,0 +1,122 @@
+================================
+Special Files and Device Numbers
+================================
+
+.. warning:: 
+    Migrated from: 
+    https://cwiki.apache.org/confluence/display/NUTTX/Special+Files+and+Device+Numbers
+
+Special Files in Unix-like File Systems
+=======================================
+
+In Unix-like operating systems, a `special file` is an interface for 
+a device driver that appears in a file system as if it were an 
+ordinary file. Special files are a feature built into Unix-like 
+file systems such as EXT2.
+
+See also `Wikipedia article <https://en.wikipedia.org/wiki/Device_file>`_.
+
+
+Special Files in Other File Systems
+===================================
+
+Some other file systems systems, such as NTFS, also have some more 
+limited and incompatible notion of special files. Others, such as 
+FAT, do not have any such concept. Unix-like environments such as 
+Cygwin that run on these other file systems have to do things a 
+little differently: The have to emulate special files using the 
+resources of the file system that they operate on. This may mean 
+creating `regular` files with special naming conventions or with 
+special content that can be used to emulate the behavior of 
+special files.
+
+Special Files and NuttX
+=======================
+
+NuttX has done things in a very different way. There are no 
+special files supported in any file system. Rather, special 
+files can exist only in the NuttX :doc:`pseudo file system <pseudofs>`. This 
+was a decision that was made in the initial design to simplify 
+things for resource limited platforms yet still provide a mostly 
+standard Unix-like/POSIX programming environment.
+
+What are the advantages of the special files in the NuttX 
+pseudo-file system? Reduce resource usage, reduced bring-up 
+requirements. What are the disadvantages? In NuttX, special 
+files can only reside in the pseudo-file system.
+
+The only other consequence that I aware of is that NuttX cannot 
+support the POSIX requirement for the ``st_dev`` field in the 
+``struct stat`` structure.
+
+
+Device Files and Device Numbers
+===============================
+
+In a Unix-like system, devices are access special device files. 
+In a Unix-like system, device files can reside in any compatible 
+file system but, by convention, are always placed in the ``/dev`` 
+directory. NuttX achieves programming compatibility with this 
+convention because the top-level, `root` file system is the 
+pseudo-file system and ``/dev`` is part of the pseudo-file system.
+
+But there is a bigger difference that this. The bigger 
+difference is how device drivers are registered and how 
+the are accessed. The primary content of the Unix-like 
+device file is simply a number, a device number, usually 
+represented as type ``dev_t``. The device number an encoded 
+that consists of a `major` device number and a `minor` device 
+number. The major device number identifies the type of 
+driver and the minor number identifies an instance of a 
+driver of that type.
+
+There is nothing special about these number from the sense of 
+a file system. Just because device file exists with a 
+certain major and minor number, that does not mean that there 
+is actually any real driver instance backing that device file 
+up. For example, you can create a device file from the Linux 
+command line like this, knowing nothing other than major and 
+minor device number:
+
+.. code-block:: C
+
+    dev_t makedev(unsigned int maj, unsigned int min);
+
+In a Unix-like system, when you try to open a device driver 
+several things must happen: The system must open the device 
+file, obtain the device major and minor number, and then 
+look up the driver instance in some internal `registry` of 
+registered device drivers. If one is found, then the system 
+can complete the open operation.
+
+So, the device number is then a `key` of some kind into a 
+`registry` of device drivers. NuttX does this very differently: 
+There are no device numbers, rather the pseudo-file system `is` 
+the device registry! In NuttX, device files cannot be created 
+by users; they can only be created by device drivers by calling 
+the following, internal interface:
+
+.. code-block:: c 
+
+    int register_driver(FAR const char *path, FAR const struct file_operations *fops, mode_t mode, FAR void *priv);
+
+The ``path`` argument determines where in the pseudo-file system the 
+device file should be placed. ``mode`` provides device file privileges. 
+The ``fops`` and ``priv`` provide the internal information for the ``registry``.
+
+So the when you open a device driver in NuttX, many fewer steps are 
+involved: The system must still open the device file, but then 
+since the pseudo-file system `is` the device registry, all of the 
+device driver information is available and ``open`` operation 
+completes with no further actions.
+
+Named Resources
+===============
+
+This use of the pseudo-file system in NuttX to manage device 
+files is consistent with a core NuttX device philosophy: The 
+NuttX VFS and the pseudo-file system in particular, are used 
+to manage all named OS resources. That applies not only to 
+device files and other special files but also to such things 
+named message queues and named semaphores (which can be found 
+in the pseudo-file system in the ``/var`` directory).