= WinFsp as an IPC Mechanism

WinFsp enables the creation of user mode file systems for Windows. At its core WinFsp is also an Inter-Process Communication (IPC) mechanism that uses the familiar file system interface for communication. This document discusses WinFsp from that viewpoint.

== Single File API Request

When a process uses the familiar file API to access a file on Windows, this API request gets packaged into an I/O Request Packet (IRP) and gets routed to the relevant File System Driver (FSD). The usual FSD's in Windows (NTFS, FastFat, etc.) will process the IRP and return a response to the process. For the remainder of this discussion, we will call this process the Originating Process (OP).

In the WinFsp case things are more complicated. WinFsp will forward IRP's to another process, which implements a user mode file system. This process will process the IRP and return a response, which WinFsp will eventually forward to the OP. We will call the process that implements the user mode file system, the File System process (FS).

In the following we will also use the notation [U] to denote user mode processing and [K] to denote kernel mode processing. Additionally because a Context Switch always goes through kernel mode, we will simplify the diagrams and omit this detail when it is not important.

Consider then what happens when an OP issues a synchronous (non-overlapped), non-cached (non-buffered) WriteFile call.

ifdef::env-browser[]
[uml,file="WinFsp-as-an-IPC-Mechanism/synchronous.png"]
--
hide footbox

participant "OP[U]" as OPU
participant "OP[K]" as OPK
participant "FS[K]" as FSK
participant "FS[U]" as FSU

activate OPU
OPU ->OPK: WriteFile
deactivate OPU
activate OPK #Salmon
OPK-->FSK: Context Switch
deactivate OPK
activate FSK #Salmon
FSK ->FSU: TRANSACT Req
deactivate FSK
activate FSU #Salmon
FSU ->FSU: Process
activate FSU
deactivate FSU
FSU ->FSK: TRANSACT Rsp
deactivate FSU
activate FSK #Salmon
FSK-->OPU: Context Switch and Return
deactivate FSK
activate OPU
note over FSK, FSU #Salmon
    Salmon color denotes WinFsp processing.
end note
--
endif::env-browser[]
ifndef::env-browser[image::WinFsp-as-an-IPC-Mechanism/synchronous.png[]]

Let us now consider what happens when an OP issues an asynchronous (overlapped), non-cached (non-buffered) WriteFile call. This scenario does not show how the OP receives the WriteFile result.

ifdef::env-browser[]
[uml,file="WinFsp-as-an-IPC-Mechanism/asynchronous.png"]
--
hide footbox

participant "OP[U]" as OPU
participant "OP[K]" as OPK
participant "FS[K]" as FSK
participant "FS[U]" as FSU

activate OPU
OPU ->OPK: WriteFile
deactivate OPU
activate OPK #Salmon
OPK ->OPU: Return
deactivate OPK
activate OPU
OPU ->OPU: Process
activate OPU
deactivate OPU
OPU-->FSK: Context Switch
deactivate OPU
activate FSK #Salmon
FSK ->FSU: TRANSACT Req
deactivate FSK
activate FSU #Salmon
FSU ->FSU: Process
activate FSU
deactivate FSU
FSU ->FSK: TRANSACT Rsp
deactivate FSU
activate FSK #Salmon
FSK-->OPU: Context Switch
deactivate FSK
activate OPU
note over FSK, FSU #Salmon
    Salmon color denotes WinFsp processing.
end note
--
endif::env-browser[]
ifndef::env-browser[image::WinFsp-as-an-IPC-Mechanism/asynchronous.png[]]

It should be noted that from the WinFsp perspective both cases look similar. WinFsp processing occurs:

- At *OP[K]* time immediately after receipt of an IRP. An IRP is said to be in the _Pending_ stage at this point.
- At *FS[K]* time after a context switch, but before the TRANSACT call. An IRP is said to be in the _Prepare_ stage at this point.
- At *FS[K]* time after the TRANSACT call. An IRP is said to be in the _Complete_ stage at this point. Upon completion of this stage the IRP will be completed and relinquished to the OS.
- AT *FS[U]* time between the two TRANSACT calls.

The TRANSACT calls are DeviceIoControl requests that the FS issues to WinFsp. A single TRANSACT call can be used to communicate a file system response and retrieve the next file system request.

## Multiple File API Requests

Let us now consider what may happen with two simultaneous API Requests from two different processes. For example, two WriteFile requests for different files.

ifdef::env-browser[]
[uml,file="WinFsp-as-an-IPC-Mechanism/multiple.png"]
--
hide footbox

participant "OP<sub>1</sub>[U]" as OP1U
participant "OP<sub>1</sub>[K]" as OP1K
participant "OP<sub>2</sub>[U]" as OP2U
participant "OP<sub>2</sub>[K]" as OP2K
participant "FS[K]" as FSK
participant "FS[U]" as FSU

activate OP1U
OP1U ->OP1K: WriteFile
deactivate OP1U
activate OP1K #Salmon
OP1K-->OP2U: Context Switch
deactivate OP1K
activate OP2U
OP2U ->OP2K: WriteFile
deactivate OP2U
activate OP2K #Salmon
OP2K-->FSK: Context Switch
deactivate OP2K
activate FSK #Salmon
FSK ->FSU: TRANSACT\nReq<sub>1</sub>
deactivate FSK
activate FSU #Salmon
FSU ->FSU: Process
activate FSU
deactivate FSU
FSU ->FSK: TRANSACT\nRsp<sub>1</sub>
deactivate FSU
activate FSK #Salmon
FSK ->FSU: TRANSACT\nReq<sub>2</sub>
deactivate FSK
activate FSU #Salmon
FSU ->FSU: Process
activate FSU
deactivate FSU
FSU ->FSK: TRANSACT\nRsp<sub>2</sub>
deactivate FSU
activate FSK #Salmon
FSK-->OP1U: Context Switch and Return
deactivate FSK
activate OP1U
OP1U ->OP1U: Process
activate OP1U
deactivate OP1U
OP1U-->OP2U: Context Switch and Return
deactivate OP1U
activate OP2U
note over FSK, FSU #Salmon
    Salmon color denotes WinFsp processing.
end note
--
endif::env-browser[]
ifndef::env-browser[image::WinFsp-as-an-IPC-Mechanism/multiple.png[]]

Notice that it is possible for the FS to process multiple file system requests without context switching.

## I/O Queues and Performance

I/O Queues are the fundamental IPC mechanism in WinFsp. The purpose of the I/O Queue is to forward an IRP from the OP to the FS and when FS processing is complete to forward the response back to the OP. I/O Queues are discussed in detail in the WinFsp design document.

WinFsp owes its excellent performance primarily to the design of the I/O Queues. I/O Queues borrow heavily from the design of I/O completion ports and schedule threads in a similar manner:

- They have a Last-In First-Out (LIFO) wait discipline.
- They limit the number of threads that can be satisfied concurrently to the number of processors.

The first property ensures that when an FS thread finishes processing a file system request, it will very likely pick up the next one from the I/O Queue without blocking and context switching to another FS thread. Minimizing context switches results in better performance.

The second property ensures that even if there are multiple file system requests waiting to be serviced in the I/O Queue, it will not schedule more thread than the number of processors. Having more than one threads scheduled on each processor is counter-productive.