diff --git a/doc/winfsp-ipc.adoc b/doc/winfsp-ipc.adoc new file mode 100644 index 00000000..12e0662e --- /dev/null +++ b/doc/winfsp-ipc.adoc @@ -0,0 +1,171 @@ += WinFsp as an IPC Mechanism + +WinFsp enables the creation of user mode file systems for Windows. At its core WinFsp is also an Inter-Process Communication (IPC) mechanism that uses the familiar file system interface for communication. This document discusses WinFsp from that viewpoint. + +== Single File API Request + +When a process uses the familiar file API to access a file on Windows, this API request gets packaged into an I/O Request Packet (IRP) and gets routed to the relevant File System Driver (FSD). The usual FSD's in Windows (NTFS, FastFat, etc.) will process the IRP and return a response to the process. For the remainder of this discussion, we will call this process the Originating Process (OP). + +In the WinFsp case things are more complicated. WinFsp will forward IRP's to another process, which implements a user mode file system. This process will process the IRP and return a response, which WinFsp will eventually forward to the OP. We will call the process that implements the user mode file system, the File System process (FS). + +In the following we will also use the notation [U] to denote user mode processing and [K] to denote kernel mode processing. Additionally because a Context Switch always goes through kernel mode, we will simplify the diagrams and omit this detail when it is not important. + +Consider then what happens when an OP issues a synchronous (non-overlapped), non-cached (non-buffered) WriteFile call. + +[uml,file="winfsp-ipc/synchronous.png"] +-- +hide footbox + +participant "OP[U]" as OPU +participant "OP[K]" as OPK +participant "FS[K]" as FSK +participant "FS[U]" as FSU + +activate OPU +OPU ->OPK: WriteFile +deactivate OPU +activate OPK #Salmon +OPK-->FSK: Context Switch +deactivate OPK +activate FSK #Salmon +FSK ->FSU: TRANSACT Req +deactivate FSK +activate FSU #Salmon +FSU ->FSU: Process +activate FSU +deactivate FSU +FSU ->FSK: TRANSACT Rsp +deactivate FSU +activate FSK #Salmon +FSK-->OPU: Context Switch and Return +deactivate FSK +activate OPU +note over FSK, FSU #Salmon + Salmon color denotes WinFsp processing. +end note +-- + +Let us now consider what happens when an OP issues an asynchronous (overlapped), non-cached (non-buffered) WriteFile call. This scenario does not show how the OP receives the WriteFile result. + +[uml,file="winfsp-ipc/asynchronous.png"] +-- +hide footbox + +participant "OP[U]" as OPU +participant "OP[K]" as OPK +participant "FS[K]" as FSK +participant "FS[U]" as FSU + +activate OPU +OPU ->OPK: WriteFile +deactivate OPU +activate OPK #Salmon +OPK ->OPU: Return +deactivate OPK +activate OPU +OPU ->OPU: Process +activate OPU +deactivate OPU +OPU-->FSK: Context Switch +deactivate OPU +activate FSK #Salmon +FSK ->FSU: TRANSACT Req +deactivate FSK +activate FSU #Salmon +FSU ->FSU: Process +activate FSU +deactivate FSU +FSU ->FSK: TRANSACT Rsp +deactivate FSU +activate FSK #Salmon +FSK-->OPU: Context Switch +deactivate FSK +activate OPU +note over FSK, FSU #Salmon + Salmon color denotes WinFsp processing. +end note +-- + +It should be noted that from the WinFsp perspective both cases look similar. WinFsp processing occurs: + +- At *OP[K]* time immediately after receipt of an IRP. An IRP is said to be in the _Pending_ stage at this point. +- At *FS[K]* time after a context switch, but before the TRANSACT call. An IRP is said to be in the _Prepare_ stage at this point. +- At *FS[K]* time after the TRANSACT call. An IRP is said to be in the _Complete_ stage at this point. Upon completion of this stage the IRP will be completed and relinquished to the OS. +- AT *FS[U]* time between the two TRANSACT calls. + +The TRANSACT calls are DeviceIoControl requests that the FS issues to WinFsp. A single TRANSACT call can be used to communicate a file system response and retrieve the next file system request. + +## Multiple File API Requests + +Let us now consider what may happen with two simultaneous API Requests from two different processes. For example, two WriteFile requests for different files. + +[uml,file="winfsp-ipc/multiple.png"] +-- +hide footbox + +participant "OP1[U]" as OP1U +participant "OP1[K]" as OP1K +participant "OP2[U]" as OP2U +participant "OP2[K]" as OP2K +participant "FS[K]" as FSK +participant "FS[U]" as FSU + +activate OP1U +OP1U ->OP1K: WriteFile +deactivate OP1U +activate OP1K #Salmon +OP1K-->OP2U: Context Switch +deactivate OP1K +activate OP2U +OP2U ->OP2K: WriteFile +deactivate OP2U +activate OP2K #Salmon +OP2K-->FSK: Context Switch +deactivate OP2K +activate FSK #Salmon +FSK ->FSU: TRANSACT\nReq1 +deactivate FSK +activate FSU #Salmon +FSU ->FSU: Process +activate FSU +deactivate FSU +FSU ->FSK: TRANSACT\nRsp1 +deactivate FSU +activate FSK #Salmon +FSK ->FSU: TRANSACT\nReq2 +deactivate FSK +activate FSU #Salmon +FSU ->FSU: Process +activate FSU +deactivate FSU +FSU ->FSK: TRANSACT\nRsp2 +deactivate FSU +activate FSK #Salmon +FSK-->OP1U: Context Switch and Return +deactivate FSK +activate OP1U +OP1U ->OP1U: Process +activate OP1U +deactivate OP1U +OP1U-->OP2U: Context Switch and Return +deactivate OP1U +activate OP2U +note over FSK, FSU #Salmon + Salmon color denotes WinFsp processing. +end note +-- + +Notice that it is possible for the FS to process multiple file system requests without context switching. + +## I/O Queues and Performance + +I/O Queues are the fundamental IPC mechanism in WinFsp. The purpose of the I/O Queue is to forward an IRP from the OP to the FS and when FS processing is complete to forward the response back to the OP. I/O Queues are discussed in detail in the WinFsp design document. + +WinFsp owes its excellent performance primarily to the design of the I/O Queues. I/O Queues borrow heavily from the design of I/O completion ports and schedule threads in a similar manner: + +- They have a Last-In First-Out (LIFO) wait discipline. +- They limit the number of threads that can be satisfied concurrently to the number of processors. + +The first property ensures that when an FS thread finishes processing a file system request, it will very likely pick up the next one from the I/O Queue without blocking and context switching to another FS thread. Minimizing context switches results in better performance. + +The second property ensures that even if there are multiple file system requests waiting to be serviced in the I/O Queue, it will not schedule more thread than the number of processors. Having more than one threads scheduled on each processor is counter-productive. diff --git a/doc/winfsp-ipc/asynchronous.png b/doc/winfsp-ipc/asynchronous.png new file mode 100644 index 00000000..ca76cad2 Binary files /dev/null and b/doc/winfsp-ipc/asynchronous.png differ diff --git a/doc/winfsp-ipc/multiple.png b/doc/winfsp-ipc/multiple.png new file mode 100644 index 00000000..276e37ed Binary files /dev/null and b/doc/winfsp-ipc/multiple.png differ diff --git a/doc/winfsp-ipc/synchronous.png b/doc/winfsp-ipc/synchronous.png new file mode 100644 index 00000000..7ce65107 Binary files /dev/null and b/doc/winfsp-ipc/synchronous.png differ