nuttx-mirror/libs
Gregory Nutt 50410cb1d5 Fix Deadloop in VFS if CONFIG_CANCELLATION_POINTS is enabled
If cancellation points are enabled, then the following logic is activated in sem_wait().  This causes ECANCELED to be returned every time that sem_wait is called.

    int sem_wait(FAR sem_t *sem)
    {
      ...

      /* sem_wait() is a cancellation point */

      if (enter_cancellation_point())
        {
    #ifdef CONFIG_CANCELLATION_POINTS
          /* If there is a pending cancellation, then do not perform
           * the wait.  Exit now with ECANCELED.
           */

          errcode = ECANCELED;
          goto errout_with_cancelpt;
    #endif
        }
      ...

Normally this works fine.  sem_wait() is the OS API called by the application and will cancel the thread just before it returns to the application.  Since it is cancellation point, it should never be called from within the OS.

There there is is one perverse cases where sem_wait() may be nested within another cancellation point.  If open() is called, it will attempt to lock a VFS data structure and will eventually call nxmutex_lock().  nxmutex_lock() waits on a semaphore:

   int nxmutex_lock(FAR mutex_t *mutex)
   {
     ...

     for (; ; )
       {
         /* Take the semaphore (perhaps waiting) */

         ret = _SEM_WAIT(&mutex->sem);
         if (ret >= 0)
           {
             mutex->holder = _SCHED_GETTID();
             break;
           }

         ret = _SEM_ERRVAL(ret);
         if (ret != -EINTR && ret != -ECANCELED)
           {
             break;
           }
       }
   ...
}

In the FLAT build, _SEM_WAIT expands to sem_wait().  That causes the error in the logic:  It should always expand to nxsem_wait().  That is because sem_wait() is cancellation point and should never be called from with the OS or the C library internally.

The failure occurs because the cancellation point logic in sem_wait() returns -ECANCELED (via _SEM_ERRVAL) because sem_wait() is nested; it needs to return the -ECANCELED error to the outermost cancellation point which is open() in this case.  Returning -ECANCELED then causes an infinite loop to occur in nxmutex_lock().

The correct behavior in this case is to call nxsem_wait() instead of sem_wait().  nxsem_wait() is identical to sem_wait() except that it is not a cancelation point.  It will return -ECANCELED if the thread is canceled, but only once.  So no infinite loop results.

In addition, an nxsem_wait() system call was added to support the call from nxmutex_lock().

This resolves Issue #9695
2023-07-07 15:34:43 +08:00
..
libc Fix Deadloop in VFS if CONFIG_CANCELLATION_POINTS is enabled 2023-07-07 15:34:43 +08:00
libdsp libdsp: Add average filter 2023-06-12 08:39:38 +02:00
libm Indent the define statement by two spaces 2023-05-21 09:52:08 -03:00
libnx Indent the define statement by two spaces 2023-05-21 09:52:08 -03:00
libxx libs/libxx: remove redundant code 2023-05-03 19:54:19 +08:00
README.txt libs/README.txt: Fix a typo 2020-01-27 08:46:46 +01:00

README
======

This directory holds NuttX libraries.  Libraries in NuttX are very special
creatures.  They have these properties:

1. They can be shared by both application logic and logic within the OS when
   using the FLAT build.

2. But in PROTECTED and KERNEL modes, they must be built differently:  The
   copies used by applications and the OS cannot be the same.  Rather,
   separate versions of libraries must be built for the kernel and for
   applications.

3. When used by the OS, some special care must be taken to assure that the
   OS logic does not disrupt the user's errno value and that the OS does
   not create inappropriate cancellation points.

   For example, sem_wait() is both a cancellation point and modifies the
   errno value.  So within the FLAT build and without kernel version for
   the PROTECTED and KERNEL builds, the special internal OS interface
   nxsem_wait() must be used.  Within libraries, the macro _SEM_WAIT()
   (as defined in include/nuttx/semaphore.h) is used instead.  The
   definition of this macro accounts for the different usage environments.

NOTE:  The libraries under libs/ build differently from other NuttX
components:  There are no build-related files in the libs/ directory; it
is simply a container for other well-known, individual library directories.
The upper level Makefile logic is aware of the libraries within the libs/
container.

The only real function of the libs/ directory is to prevent the top-level
directory from becoming cluttered with individual libraries.