diff options
Diffstat (limited to 'Documentation/filesystems')
51 files changed, 4680 insertions, 4010 deletions
diff --git a/Documentation/filesystems/api-summary.rst b/Documentation/filesystems/api-summary.rst index aa51ffcfa029..bbb0c1c0e5cf 100644 --- a/Documentation/filesystems/api-summary.rst +++ b/Documentation/filesystems/api-summary.rst @@ -89,9 +89,6 @@ Other Functions .. kernel-doc:: fs/direct-io.c :export: -.. kernel-doc:: fs/file_table.c - :export: - .. kernel-doc:: fs/libfs.c :export: diff --git a/Documentation/filesystems/binderfs.rst b/Documentation/filesystems/binderfs.rst deleted file mode 100644 index c009671f8434..000000000000 --- a/Documentation/filesystems/binderfs.rst +++ /dev/null @@ -1,68 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 - -The Android binderfs Filesystem -=============================== - -Android binderfs is a filesystem for the Android binder IPC mechanism. It -allows to dynamically add and remove binder devices at runtime. Binder devices -located in a new binderfs instance are independent of binder devices located in -other binderfs instances. Mounting a new binderfs instance makes it possible -to get a set of private binder devices. - -Mounting binderfs ------------------ - -Android binderfs can be mounted with:: - - mkdir /dev/binderfs - mount -t binder binder /dev/binderfs - -at which point a new instance of binderfs will show up at ``/dev/binderfs``. -In a fresh instance of binderfs no binder devices will be present. There will -only be a ``binder-control`` device which serves as the request handler for -binderfs. Mounting another binderfs instance at a different location will -create a new and separate instance from all other binderfs mounts. This is -identical to the behavior of e.g. ``devpts`` and ``tmpfs``. The Android -binderfs filesystem can be mounted in user namespaces. - -Options -------- -max - binderfs instances can be mounted with a limit on the number of binder - devices that can be allocated. The ``max=<count>`` mount option serves as - a per-instance limit. If ``max=<count>`` is set then only ``<count>`` number - of binder devices can be allocated in this binderfs instance. - -Allocating binder Devices -------------------------- - -.. _ioctl: http://man7.org/linux/man-pages/man2/ioctl.2.html - -To allocate a new binder device in a binderfs instance a request needs to be -sent through the ``binder-control`` device node. A request is sent in the form -of an `ioctl() <ioctl_>`_. - -What a program needs to do is to open the ``binder-control`` device node and -send a ``BINDER_CTL_ADD`` request to the kernel. Users of binderfs need to -tell the kernel which name the new binder device should get. By default a name -can only contain up to ``BINDERFS_MAX_NAME`` chars including the terminating -zero byte. - -Once the request is made via an `ioctl() <ioctl_>`_ passing a ``struct -binder_device`` with the name to the kernel it will allocate a new binder -device and return the major and minor number of the new device in the struct -(This is necessary because binderfs allocates a major device number -dynamically.). After the `ioctl() <ioctl_>`_ returns there will be a new -binder device located under /dev/binderfs with the chosen name. - -Deleting binder Devices ------------------------ - -.. _unlink: http://man7.org/linux/man-pages/man2/unlink.2.html -.. _rm: http://man7.org/linux/man-pages/man1/rm.1.html - -Binderfs binder devices can be deleted via `unlink() <unlink_>`_. This means -that the `rm() <rm_>`_ tool can be used to delete them. Note that the -``binder-control`` device cannot be deleted since this would make the binderfs -instance unuseable. The ``binder-control`` device will be deleted when the -binderfs instance is unmounted and all references to it have been dropped. diff --git a/Documentation/filesystems/ceph.txt b/Documentation/filesystems/ceph.txt index d2c6a5ccf0f5..b19b6a03f91c 100644 --- a/Documentation/filesystems/ceph.txt +++ b/Documentation/filesystems/ceph.txt @@ -158,6 +158,20 @@ Mount Options copies. Currently, it's only used in copy_file_range, which will revert to the default VFS implementation if this option is used. + recover_session=<no|clean> + Set auto reconnect mode in the case where the client is blacklisted. The + available modes are "no" and "clean". The default is "no". + + * no: never attempt to reconnect when client detects that it has been + blacklisted. Operations will generally fail after being blacklisted. + + * clean: client reconnects to the ceph cluster automatically when it + detects that it has been blacklisted. During reconnect, client drops + dirty data/metadata, invalidates page caches and writable file handles. + After reconnect, file locks become stale because the MDS loses track + of them. If an inode contains any stale file locks, read/write on the + inode is not allowed until applications release all stale file locks. + More Information ================ diff --git a/Documentation/filesystems/cifs/AUTHORS b/Documentation/filesystems/cifs/AUTHORS deleted file mode 100644 index 75865da2ce14..000000000000 --- a/Documentation/filesystems/cifs/AUTHORS +++ /dev/null @@ -1,63 +0,0 @@ -Original Author -=============== -Steve French (sfrench@samba.org) - -The author wishes to express his appreciation and thanks to: -Andrew Tridgell (Samba team) for his early suggestions about smb/cifs VFS -improvements. Thanks to IBM for allowing me time and test resources to pursue -this project, to Jim McDonough from IBM (and the Samba Team) for his help, to -the IBM Linux JFS team for explaining many esoteric Linux filesystem features. -Jeremy Allison of the Samba team has done invaluable work in adding the server -side of the original CIFS Unix extensions and reviewing and implementing -portions of the newer CIFS POSIX extensions into the Samba 3 file server. Thank -Dave Boutcher of IBM Rochester (author of the OS/400 smb/cifs filesystem client) -for proving years ago that very good smb/cifs clients could be done on Unix-like -operating systems. Volker Lendecke, Andrew Tridgell, Urban Widmark, John -Newbigin and others for their work on the Linux smbfs module. Thanks to -the other members of the Storage Network Industry Association CIFS Technical -Workgroup for their work specifying this highly complex protocol and finally -thanks to the Samba team for their technical advice and encouragement. - -Patch Contributors ------------------- -Zwane Mwaikambo -Andi Kleen -Amrut Joshi -Shobhit Dayal -Sergey Vlasov -Richard Hughes -Yury Umanets -Mark Hamzy (for some of the early cifs IPv6 work) -Domen Puncer -Jesper Juhl (in particular for lots of whitespace/formatting cleanup) -Vince Negri and Dave Stahl (for finding an important caching bug) -Adrian Bunk (kcalloc cleanups) -Miklos Szeredi -Kazeon team for various fixes especially for 2.4 version. -Asser Ferno (Change Notify support) -Shaggy (Dave Kleikamp) for innumerable small fs suggestions and some good cleanup -Gunter Kukkukk (testing and suggestions for support of old servers) -Igor Mammedov (DFS support) -Jeff Layton (many, many fixes, as well as great work on the cifs Kerberos code) -Scott Lovenberg -Pavel Shilovsky (for great work adding SMB2 support, and various SMB3 features) -Aurelien Aptel (for DFS SMB3 work and some key bug fixes) -Ronnie Sahlberg (for SMB3 xattr work, bug fixes, and lots of great work on compounding) -Shirish Pargaonkar (for many ACL patches over the years) -Sachin Prabhu (many bug fixes, including for reconnect, copy offload and security) -Paulo Alcantara -Long Li (some great work on RDMA, SMB Direct) - - -Test case and Bug Report contributors -------------------------------------- -Thanks to those in the community who have submitted detailed bug reports -and debug of problems they have found: Jochen Dolze, David Blaine, -Rene Scharfe, Martin Josefsson, Alexander Wild, Anthony Liguori, -Lars Muller, Urban Widmark, Massimiliano Ferrero, Howard Owen, -Olaf Kirch, Kieron Briggs, Nick Millington and others. Also special -mention to the Stanford Checker (SWAT) which pointed out many minor -bugs in error paths. Valuable suggestions also have come from Al Viro -and Dave Miller. - -And thanks to the IBM LTC and Power test teams and SuSE and Citrix and RedHat testers for finding multiple bugs during excellent stress test runs. diff --git a/Documentation/filesystems/cifs/CHANGES b/Documentation/filesystems/cifs/CHANGES deleted file mode 100644 index 1df7f4910eb2..000000000000 --- a/Documentation/filesystems/cifs/CHANGES +++ /dev/null @@ -1,4 +0,0 @@ -See https://wiki.samba.org/index.php/LinuxCIFSKernel for summary -information (that may be easier to read than parsing the output of -"git log fs/cifs") about fixes/improvements to CIFS/SMB2/SMB3 support (changes -to cifs.ko module) by kernel version (and cifs internal module version). diff --git a/Documentation/filesystems/cifs/README b/Documentation/filesystems/cifs/README deleted file mode 100644 index 4a804619cff2..000000000000 --- a/Documentation/filesystems/cifs/README +++ /dev/null @@ -1,743 +0,0 @@ -This module supports the SMB3 family of advanced network protocols (as well -as older dialects, originally called "CIFS" or SMB1). - -The CIFS VFS module for Linux supports many advanced network filesystem -features such as hierarchical DFS like namespace, hardlinks, locking and more. -It was designed to comply with the SNIA CIFS Technical Reference (which -supersedes the 1992 X/Open SMB Standard) as well as to perform best practice -practical interoperability with Windows 2000, Windows XP, Samba and equivalent -servers. This code was developed in participation with the Protocol Freedom -Information Foundation. CIFS and now SMB3 has now become a defacto -standard for interoperating between Macs and Windows and major NAS appliances. - -Please see - MS-SMB2 (for detailed SMB2/SMB3/SMB3.1.1 protocol specification) - http://protocolfreedom.org/ and - http://samba.org/samba/PFIF/ -for more details. - - -For questions or bug reports please contact: - smfrench@gmail.com - -See the project page at: https://wiki.samba.org/index.php/LinuxCIFS_utils - -Build instructions: -================== -For Linux: -1) Download the kernel (e.g. from http://www.kernel.org) -and change directory into the top of the kernel directory tree -(e.g. /usr/src/linux-2.5.73) -2) make menuconfig (or make xconfig) -3) select cifs from within the network filesystem choices -4) save and exit -5) make - - -Installation instructions: -========================= -If you have built the CIFS vfs as module (successfully) simply -type "make modules_install" (or if you prefer, manually copy the file to -the modules directory e.g. /lib/modules/2.4.10-4GB/kernel/fs/cifs/cifs.ko). - -If you have built the CIFS vfs into the kernel itself, follow the instructions -for your distribution on how to install a new kernel (usually you -would simply type "make install"). - -If you do not have the utility mount.cifs (in the Samba 4.x source tree and on -the CIFS VFS web site) copy it to the same directory in which mount helpers -reside (usually /sbin). Although the helper software is not -required, mount.cifs is recommended. Most distros include a "cifs-utils" -package that includes this utility so it is recommended to install this. - -Note that running the Winbind pam/nss module (logon service) on all of your -Linux clients is useful in mapping Uids and Gids consistently across the -domain to the proper network user. The mount.cifs mount helper can be -found at cifs-utils.git on git.samba.org - -If cifs is built as a module, then the size and number of network buffers -and maximum number of simultaneous requests to one server can be configured. -Changing these from their defaults is not recommended. By executing modinfo - modinfo kernel/fs/cifs/cifs.ko -on kernel/fs/cifs/cifs.ko the list of configuration changes that can be made -at module initialization time (by running insmod cifs.ko) can be seen. - -Recommendations -=============== -To improve security the SMB2.1 dialect or later (usually will get SMB3) is now -the new default. To use old dialects (e.g. to mount Windows XP) use "vers=1.0" -on mount (or vers=2.0 for Windows Vista). Note that the CIFS (vers=1.0) is -much older and less secure than the default dialect SMB3 which includes -many advanced security features such as downgrade attack detection -and encrypted shares and stronger signing and authentication algorithms. -There are additional mount options that may be helpful for SMB3 to get -improved POSIX behavior (NB: can use vers=3.0 to force only SMB3, never 2.1): - "mfsymlinks" and "cifsacl" and "idsfromsid" - -Allowing User Mounts -==================== -To permit users to mount and unmount over directories they own is possible -with the cifs vfs. A way to enable such mounting is to mark the mount.cifs -utility as suid (e.g. "chmod +s /sbin/mount.cifs). To enable users to -umount shares they mount requires -1) mount.cifs version 1.4 or later -2) an entry for the share in /etc/fstab indicating that a user may -unmount it e.g. -//server/usersharename /mnt/username cifs user 0 0 - -Note that when the mount.cifs utility is run suid (allowing user mounts), -in order to reduce risks, the "nosuid" mount flag is passed in on mount to -disallow execution of an suid program mounted on the remote target. -When mount is executed as root, nosuid is not passed in by default, -and execution of suid programs on the remote target would be enabled -by default. This can be changed, as with nfs and other filesystems, -by simply specifying "nosuid" among the mount options. For user mounts -though to be able to pass the suid flag to mount requires rebuilding -mount.cifs with the following flag: CIFS_ALLOW_USR_SUID - -There is a corresponding manual page for cifs mounting in the Samba 3.0 and -later source tree in docs/manpages/mount.cifs.8 - -Allowing User Unmounts -====================== -To permit users to ummount directories that they have user mounted (see above), -the utility umount.cifs may be used. It may be invoked directly, or if -umount.cifs is placed in /sbin, umount can invoke the cifs umount helper -(at least for most versions of the umount utility) for umount of cifs -mounts, unless umount is invoked with -i (which will avoid invoking a umount -helper). As with mount.cifs, to enable user unmounts umount.cifs must be marked -as suid (e.g. "chmod +s /sbin/umount.cifs") or equivalent (some distributions -allow adding entries to a file to the /etc/permissions file to achieve the -equivalent suid effect). For this utility to succeed the target path -must be a cifs mount, and the uid of the current user must match the uid -of the user who mounted the resource. - -Also note that the customary way of allowing user mounts and unmounts is -(instead of using mount.cifs and unmount.cifs as suid) to add a line -to the file /etc/fstab for each //server/share you wish to mount, but -this can become unwieldy when potential mount targets include many -or unpredictable UNC names. - -Samba Considerations -==================== -Most current servers support SMB2.1 and SMB3 which are more secure, -but there are useful protocol extensions for the older less secure CIFS -dialect, so to get the maximum benefit if mounting using the older dialect -(CIFS/SMB1), we recommend using a server that supports the SNIA CIFS -Unix Extensions standard (e.g. almost any version of Samba ie version -2.2.5 or later) but the CIFS vfs works fine with a wide variety of CIFS servers. -Note that uid, gid and file permissions will display default values if you do -not have a server that supports the Unix extensions for CIFS (such as Samba -2.2.5 or later). To enable the Unix CIFS Extensions in the Samba server, add -the line: - - unix extensions = yes - -to your smb.conf file on the server. Note that the following smb.conf settings -are also useful (on the Samba server) when the majority of clients are Unix or -Linux: - - case sensitive = yes - delete readonly = yes - ea support = yes - -Note that server ea support is required for supporting xattrs from the Linux -cifs client, and that EA support is present in later versions of Samba (e.g. -3.0.6 and later (also EA support works in all versions of Windows, at least to -shares on NTFS filesystems). Extended Attribute (xattr) support is an optional -feature of most Linux filesystems which may require enabling via -make menuconfig. Client support for extended attributes (user xattr) can be -disabled on a per-mount basis by specifying "nouser_xattr" on mount. - -The CIFS client can get and set POSIX ACLs (getfacl, setfacl) to Samba servers -version 3.10 and later. Setting POSIX ACLs requires enabling both XATTR and -then POSIX support in the CIFS configuration options when building the cifs -module. POSIX ACL support can be disabled on a per mount basic by specifying -"noacl" on mount. - -Some administrators may want to change Samba's smb.conf "map archive" and -"create mask" parameters from the default. Unless the create mask is changed -newly created files can end up with an unnecessarily restrictive default mode, -which may not be what you want, although if the CIFS Unix extensions are -enabled on the server and client, subsequent setattr calls (e.g. chmod) can -fix the mode. Note that creating special devices (mknod) remotely -may require specifying a mkdev function to Samba if you are not using -Samba 3.0.6 or later. For more information on these see the manual pages -("man smb.conf") on the Samba server system. Note that the cifs vfs, -unlike the smbfs vfs, does not read the smb.conf on the client system -(the few optional settings are passed in on mount via -o parameters instead). -Note that Samba 2.2.7 or later includes a fix that allows the CIFS VFS to delete -open files (required for strict POSIX compliance). Windows Servers already -supported this feature. Samba server does not allow symlinks that refer to files -outside of the share, so in Samba versions prior to 3.0.6, most symlinks to -files with absolute paths (ie beginning with slash) such as: - ln -s /mnt/foo bar -would be forbidden. Samba 3.0.6 server or later includes the ability to create -such symlinks safely by converting unsafe symlinks (ie symlinks to server -files that are outside of the share) to a samba specific format on the server -that is ignored by local server applications and non-cifs clients and that will -not be traversed by the Samba server). This is opaque to the Linux client -application using the cifs vfs. Absolute symlinks will work to Samba 3.0.5 or -later, but only for remote clients using the CIFS Unix extensions, and will -be invisbile to Windows clients and typically will not affect local -applications running on the same server as Samba. - -Use instructions: -================ -Once the CIFS VFS support is built into the kernel or installed as a module -(cifs.ko), you can use mount syntax like the following to access Samba or -Mac or Windows servers: - - mount -t cifs //9.53.216.11/e$ /mnt -o username=myname,password=mypassword - -Before -o the option -v may be specified to make the mount.cifs -mount helper display the mount steps more verbosely. -After -o the following commonly used cifs vfs specific options -are supported: - - username=<username> - password=<password> - domain=<domain name> - -Other cifs mount options are described below. Use of TCP names (in addition to -ip addresses) is available if the mount helper (mount.cifs) is installed. If -you do not trust the server to which are mounted, or if you do not have -cifs signing enabled (and the physical network is insecure), consider use -of the standard mount options "noexec" and "nosuid" to reduce the risk of -running an altered binary on your local system (downloaded from a hostile server -or altered by a hostile router). - -Although mounting using format corresponding to the CIFS URL specification is -not possible in mount.cifs yet, it is possible to use an alternate format -for the server and sharename (which is somewhat similar to NFS style mount -syntax) instead of the more widely used UNC format (i.e. \\server\share): - mount -t cifs tcp_name_of_server:share_name /mnt -o user=myname,pass=mypasswd - -When using the mount helper mount.cifs, passwords may be specified via alternate -mechanisms, instead of specifying it after -o using the normal "pass=" syntax -on the command line: -1) By including it in a credential file. Specify credentials=filename as one -of the mount options. Credential files contain two lines - username=someuser - password=your_password -2) By specifying the password in the PASSWD environment variable (similarly -the user name can be taken from the USER environment variable). -3) By specifying the password in a file by name via PASSWD_FILE -4) By specifying the password in a file by file descriptor via PASSWD_FD - -If no password is provided, mount.cifs will prompt for password entry - -Restrictions -============ -Servers must support either "pure-TCP" (port 445 TCP/IP CIFS connections) or RFC -1001/1002 support for "Netbios-Over-TCP/IP." This is not likely to be a -problem as most servers support this. - -Valid filenames differ between Windows and Linux. Windows typically restricts -filenames which contain certain reserved characters (e.g.the character : -which is used to delimit the beginning of a stream name by Windows), while -Linux allows a slightly wider set of valid characters in filenames. Windows -servers can remap such characters when an explicit mapping is specified in -the Server's registry. Samba starting with version 3.10 will allow such -filenames (ie those which contain valid Linux characters, which normally -would be forbidden for Windows/CIFS semantics) as long as the server is -configured for Unix Extensions (and the client has not disabled -/proc/fs/cifs/LinuxExtensionsEnabled). In addition the mount option -"mapposix" can be used on CIFS (vers=1.0) to force the mapping of -illegal Windows/NTFS/SMB characters to a remap range (this mount parm -is the default for SMB3). This remap ("mapposix") range is also -compatible with Mac (and "Services for Mac" on some older Windows). - -CIFS VFS Mount Options -====================== -A partial list of the supported mount options follows: - username The user name to use when trying to establish - the CIFS session. - password The user password. If the mount helper is - installed, the user will be prompted for password - if not supplied. - ip The ip address of the target server - unc The target server Universal Network Name (export) to - mount. - domain Set the SMB/CIFS workgroup name prepended to the - username during CIFS session establishment - forceuid Set the default uid for inodes to the uid - passed in on mount. For mounts to servers - which do support the CIFS Unix extensions, such as a - properly configured Samba server, the server provides - the uid, gid and mode so this parameter should not be - specified unless the server and clients uid and gid - numbering differ. If the server and client are in the - same domain (e.g. running winbind or nss_ldap) and - the server supports the Unix Extensions then the uid - and gid can be retrieved from the server (and uid - and gid would not have to be specified on the mount. - For servers which do not support the CIFS Unix - extensions, the default uid (and gid) returned on lookup - of existing files will be the uid (gid) of the person - who executed the mount (root, except when mount.cifs - is configured setuid for user mounts) unless the "uid=" - (gid) mount option is specified. Also note that permission - checks (authorization checks) on accesses to a file occur - at the server, but there are cases in which an administrator - may want to restrict at the client as well. For those - servers which do not report a uid/gid owner - (such as Windows), permissions can also be checked at the - client, and a crude form of client side permission checking - can be enabled by specifying file_mode and dir_mode on - the client. (default) - forcegid (similar to above but for the groupid instead of uid) (default) - noforceuid Fill in file owner information (uid) by requesting it from - the server if possible. With this option, the value given in - the uid= option (on mount) will only be used if the server - can not support returning uids on inodes. - noforcegid (similar to above but for the group owner, gid, instead of uid) - uid Set the default uid for inodes, and indicate to the - cifs kernel driver which local user mounted. If the server - supports the unix extensions the default uid is - not used to fill in the owner fields of inodes (files) - unless the "forceuid" parameter is specified. - gid Set the default gid for inodes (similar to above). - file_mode If CIFS Unix extensions are not supported by the server - this overrides the default mode for file inodes. - fsc Enable local disk caching using FS-Cache (off by default). This - option could be useful to improve performance on a slow link, - heavily loaded server and/or network where reading from the - disk is faster than reading from the server (over the network). - This could also impact scalability positively as the - number of calls to the server are reduced. However, local - caching is not suitable for all workloads for e.g. read-once - type workloads. So, you need to consider carefully your - workload/scenario before using this option. Currently, local - disk caching is functional for CIFS files opened as read-only. - dir_mode If CIFS Unix extensions are not supported by the server - this overrides the default mode for directory inodes. - port attempt to contact the server on this tcp port, before - trying the usual ports (port 445, then 139). - iocharset Codepage used to convert local path names to and from - Unicode. Unicode is used by default for network path - names if the server supports it. If iocharset is - not specified then the nls_default specified - during the local client kernel build will be used. - If server does not support Unicode, this parameter is - unused. - rsize default read size (usually 16K). The client currently - can not use rsize larger than CIFSMaxBufSize. CIFSMaxBufSize - defaults to 16K and may be changed (from 8K to the maximum - kmalloc size allowed by your kernel) at module install time - for cifs.ko. Setting CIFSMaxBufSize to a very large value - will cause cifs to use more memory and may reduce performance - in some cases. To use rsize greater than 127K (the original - cifs protocol maximum) also requires that the server support - a new Unix Capability flag (for very large read) which some - newer servers (e.g. Samba 3.0.26 or later) do. rsize can be - set from a minimum of 2048 to a maximum of 130048 (127K or - CIFSMaxBufSize, whichever is smaller) - wsize default write size (default 57344) - maximum wsize currently allowed by CIFS is 57344 (fourteen - 4096 byte pages) - actimeo=n attribute cache timeout in seconds (default 1 second). - After this timeout, the cifs client requests fresh attribute - information from the server. This option allows to tune the - attribute cache timeout to suit the workload needs. Shorter - timeouts mean better the cache coherency, but increased number - of calls to the server. Longer timeouts mean reduced number - of calls to the server at the expense of less stricter cache - coherency checks (i.e. incorrect attribute cache for a short - period of time). - rw mount the network share read-write (note that the - server may still consider the share read-only) - ro mount network share read-only - version used to distinguish different versions of the - mount helper utility (not typically needed) - sep if first mount option (after the -o), overrides - the comma as the separator between the mount - parms. e.g. - -o user=myname,password=mypassword,domain=mydom - could be passed instead with period as the separator by - -o sep=.user=myname.password=mypassword.domain=mydom - this might be useful when comma is contained within username - or password or domain. This option is less important - when the cifs mount helper cifs.mount (version 1.1 or later) - is used. - nosuid Do not allow remote executables with the suid bit - program to be executed. This is only meaningful for mounts - to servers such as Samba which support the CIFS Unix Extensions. - If you do not trust the servers in your network (your mount - targets) it is recommended that you specify this option for - greater security. - exec Permit execution of binaries on the mount. - noexec Do not permit execution of binaries on the mount. - dev Recognize block devices on the remote mount. - nodev Do not recognize devices on the remote mount. - suid Allow remote files on this mountpoint with suid enabled to - be executed (default for mounts when executed as root, - nosuid is default for user mounts). - credentials Although ignored by the cifs kernel component, it is used by - the mount helper, mount.cifs. When mount.cifs is installed it - opens and reads the credential file specified in order - to obtain the userid and password arguments which are passed to - the cifs vfs. - guest Although ignored by the kernel component, the mount.cifs - mount helper will not prompt the user for a password - if guest is specified on the mount options. If no - password is specified a null password will be used. - perm Client does permission checks (vfs_permission check of uid - and gid of the file against the mode and desired operation), - Note that this is in addition to the normal ACL check on the - target machine done by the server software. - Client permission checking is enabled by default. - noperm Client does not do permission checks. This can expose - files on this mount to access by other users on the local - client system. It is typically only needed when the server - supports the CIFS Unix Extensions but the UIDs/GIDs on the - client and server system do not match closely enough to allow - access by the user doing the mount, but it may be useful with - non CIFS Unix Extension mounts for cases in which the default - mode is specified on the mount but is not to be enforced on the - client (e.g. perhaps when MultiUserMount is enabled) - Note that this does not affect the normal ACL check on the - target machine done by the server software (of the server - ACL against the user name provided at mount time). - serverino Use server's inode numbers instead of generating automatically - incrementing inode numbers on the client. Although this will - make it easier to spot hardlinked files (as they will have - the same inode numbers) and inode numbers may be persistent, - note that the server does not guarantee that the inode numbers - are unique if multiple server side mounts are exported under a - single share (since inode numbers on the servers might not - be unique if multiple filesystems are mounted under the same - shared higher level directory). Note that some older - (e.g. pre-Windows 2000) do not support returning UniqueIDs - or the CIFS Unix Extensions equivalent and for those - this mount option will have no effect. Exporting cifs mounts - under nfsd requires this mount option on the cifs mount. - This is now the default if server supports the - required network operation. - noserverino Client generates inode numbers (rather than using the actual one - from the server). These inode numbers will vary after - unmount or reboot which can confuse some applications, - but not all server filesystems support unique inode - numbers. - setuids If the CIFS Unix extensions are negotiated with the server - the client will attempt to set the effective uid and gid of - the local process on newly created files, directories, and - devices (create, mkdir, mknod). If the CIFS Unix Extensions - are not negotiated, for newly created files and directories - instead of using the default uid and gid specified on - the mount, cache the new file's uid and gid locally which means - that the uid for the file can change when the inode is - reloaded (or the user remounts the share). - nosetuids The client will not attempt to set the uid and gid on - on newly created files, directories, and devices (create, - mkdir, mknod) which will result in the server setting the - uid and gid to the default (usually the server uid of the - user who mounted the share). Letting the server (rather than - the client) set the uid and gid is the default. If the CIFS - Unix Extensions are not negotiated then the uid and gid for - new files will appear to be the uid (gid) of the mounter or the - uid (gid) parameter specified on the mount. - netbiosname When mounting to servers via port 139, specifies the RFC1001 - source name to use to represent the client netbios machine - name when doing the RFC1001 netbios session initialize. - direct Do not do inode data caching on files opened on this mount. - This precludes mmapping files on this mount. In some cases - with fast networks and little or no caching benefits on the - client (e.g. when the application is doing large sequential - reads bigger than page size without rereading the same data) - this can provide better performance than the default - behavior which caches reads (readahead) and writes - (writebehind) through the local Linux client pagecache - if oplock (caching token) is granted and held. Note that - direct allows write operations larger than page size - to be sent to the server. - strictcache Use for switching on strict cache mode. In this mode the - client read from the cache all the time it has Oplock Level II, - otherwise - read from the server. All written data are stored - in the cache, but if the client doesn't have Exclusive Oplock, - it writes the data to the server. - rwpidforward Forward pid of a process who opened a file to any read or write - operation on that file. This prevent applications like WINE - from failing on read and write if we use mandatory brlock style. - acl Allow setfacl and getfacl to manage posix ACLs if server - supports them. (default) - noacl Do not allow setfacl and getfacl calls on this mount - user_xattr Allow getting and setting user xattrs (those attributes whose - name begins with "user." or "os2.") as OS/2 EAs (extended - attributes) to the server. This allows support of the - setfattr and getfattr utilities. (default) - nouser_xattr Do not allow getfattr/setfattr to get/set/list xattrs - mapchars Translate six of the seven reserved characters (not backslash) - *?<>|: - to the remap range (above 0xF000), which also - allows the CIFS client to recognize files created with - such characters by Windows's POSIX emulation. This can - also be useful when mounting to most versions of Samba - (which also forbids creating and opening files - whose names contain any of these seven characters). - This has no effect if the server does not support - Unicode on the wire. - nomapchars Do not translate any of these seven characters (default). - nocase Request case insensitive path name matching (case - sensitive is the default if the server supports it). - (mount option "ignorecase" is identical to "nocase") - posixpaths If CIFS Unix extensions are supported, attempt to - negotiate posix path name support which allows certain - characters forbidden in typical CIFS filenames, without - requiring remapping. (default) - noposixpaths If CIFS Unix extensions are supported, do not request - posix path name support (this may cause servers to - reject creatingfile with certain reserved characters). - nounix Disable the CIFS Unix Extensions for this mount (tree - connection). This is rarely needed, but it may be useful - in order to turn off multiple settings all at once (ie - posix acls, posix locks, posix paths, symlink support - and retrieving uids/gids/mode from the server) or to - work around a bug in server which implement the Unix - Extensions. - nobrl Do not send byte range lock requests to the server. - This is necessary for certain applications that break - with cifs style mandatory byte range locks (and most - cifs servers do not yet support requesting advisory - byte range locks). - forcemandatorylock Even if the server supports posix (advisory) byte range - locking, send only mandatory lock requests. For some - (presumably rare) applications, originally coded for - DOS/Windows, which require Windows style mandatory byte range - locking, they may be able to take advantage of this option, - forcing the cifs client to only send mandatory locks - even if the cifs server would support posix advisory locks. - "forcemand" is accepted as a shorter form of this mount - option. - nostrictsync If this mount option is set, when an application does an - fsync call then the cifs client does not send an SMB Flush - to the server (to force the server to write all dirty data - for this file immediately to disk), although cifs still sends - all dirty (cached) file data to the server and waits for the - server to respond to the write. Since SMB Flush can be - very slow, and some servers may be reliable enough (to risk - delaying slightly flushing the data to disk on the server), - turning on this option may be useful to improve performance for - applications that fsync too much, at a small risk of server - crash. If this mount option is not set, by default cifs will - send an SMB flush request (and wait for a response) on every - fsync call. - nodfs Disable DFS (global name space support) even if the - server claims to support it. This can help work around - a problem with parsing of DFS paths with Samba server - versions 3.0.24 and 3.0.25. - remount remount the share (often used to change from ro to rw mounts - or vice versa) - cifsacl Report mode bits (e.g. on stat) based on the Windows ACL for - the file. (EXPERIMENTAL) - servern Specify the server 's netbios name (RFC1001 name) to use - when attempting to setup a session to the server. - This is needed for mounting to some older servers (such - as OS/2 or Windows 98 and Windows ME) since they do not - support a default server name. A server name can be up - to 15 characters long and is usually uppercased. - sfu When the CIFS Unix Extensions are not negotiated, attempt to - create device files and fifos in a format compatible with - Services for Unix (SFU). In addition retrieve bits 10-12 - of the mode via the SETFILEBITS extended attribute (as - SFU does). In the future the bottom 9 bits of the - mode also will be emulated using queries of the security - descriptor (ACL). - mfsymlinks Enable support for Minshall+French symlinks - (see http://wiki.samba.org/index.php/UNIX_Extensions#Minshall.2BFrench_symlinks) - This option is ignored when specified together with the - 'sfu' option. Minshall+French symlinks are used even if - the server supports the CIFS Unix Extensions. - sign Must use packet signing (helps avoid unwanted data modification - by intermediate systems in the route). Note that signing - does not work with lanman or plaintext authentication. - seal Must seal (encrypt) all data on this mounted share before - sending on the network. Requires support for Unix Extensions. - Note that this differs from the sign mount option in that it - causes encryption of data sent over this mounted share but other - shares mounted to the same server are unaffected. - locallease This option is rarely needed. Fcntl F_SETLEASE is - used by some applications such as Samba and NFSv4 server to - check to see whether a file is cacheable. CIFS has no way - to explicitly request a lease, but can check whether a file - is cacheable (oplocked). Unfortunately, even if a file - is not oplocked, it could still be cacheable (ie cifs client - could grant fcntl leases if no other local processes are using - the file) for cases for example such as when the server does not - support oplocks and the user is sure that the only updates to - the file will be from this client. Specifying this mount option - will allow the cifs client to check for leases (only) locally - for files which are not oplocked instead of denying leases - in that case. (EXPERIMENTAL) - sec Security mode. Allowed values are: - none attempt to connection as a null user (no name) - krb5 Use Kerberos version 5 authentication - krb5i Use Kerberos authentication and packet signing - ntlm Use NTLM password hashing (default) - ntlmi Use NTLM password hashing with signing (if - /proc/fs/cifs/PacketSigningEnabled on or if - server requires signing also can be the default) - ntlmv2 Use NTLMv2 password hashing - ntlmv2i Use NTLMv2 password hashing with packet signing - lanman (if configured in kernel config) use older - lanman hash -hard Retry file operations if server is not responding -soft Limit retries to unresponsive servers (usually only - one retry) before returning an error. (default) - -The mount.cifs mount helper also accepts a few mount options before -o -including: - - -S take password from stdin (equivalent to setting the environment - variable "PASSWD_FD=0" - -V print mount.cifs version - -? display simple usage information - -With most 2.6 kernel versions of modutils, the version of the cifs kernel -module can be displayed via modinfo. - -Misc /proc/fs/cifs Flags and Debug Info -======================================= -Informational pseudo-files: -DebugData Displays information about active CIFS sessions and - shares, features enabled as well as the cifs.ko - version. -Stats Lists summary resource usage information as well as per - share statistics. - -Configuration pseudo-files: -SecurityFlags Flags which control security negotiation and - also packet signing. Authentication (may/must) - flags (e.g. for NTLM and/or NTLMv2) may be combined with - the signing flags. Specifying two different password - hashing mechanisms (as "must use") on the other hand - does not make much sense. Default flags are - 0x07007 - (NTLM, NTLMv2 and packet signing allowed). The maximum - allowable flags if you want to allow mounts to servers - using weaker password hashes is 0x37037 (lanman, - plaintext, ntlm, ntlmv2, signing allowed). Some - SecurityFlags require the corresponding menuconfig - options to be enabled (lanman and plaintext require - CONFIG_CIFS_WEAK_PW_HASH for example). Enabling - plaintext authentication currently requires also - enabling lanman authentication in the security flags - because the cifs module only supports sending - laintext passwords using the older lanman dialect - form of the session setup SMB. (e.g. for authentication - using plain text passwords, set the SecurityFlags - to 0x30030): - - may use packet signing 0x00001 - must use packet signing 0x01001 - may use NTLM (most common password hash) 0x00002 - must use NTLM 0x02002 - may use NTLMv2 0x00004 - must use NTLMv2 0x04004 - may use Kerberos security 0x00008 - must use Kerberos 0x08008 - may use lanman (weak) password hash 0x00010 - must use lanman password hash 0x10010 - may use plaintext passwords 0x00020 - must use plaintext passwords 0x20020 - (reserved for future packet encryption) 0x00040 - -cifsFYI If set to non-zero value, additional debug information - will be logged to the system error log. This field - contains three flags controlling different classes of - debugging entries. The maximum value it can be set - to is 7 which enables all debugging points (default 0). - Some debugging statements are not compiled into the - cifs kernel unless CONFIG_CIFS_DEBUG2 is enabled in the - kernel configuration. cifsFYI may be set to one or - nore of the following flags (7 sets them all): - - log cifs informational messages 0x01 - log return codes from cifs entry points 0x02 - log slow responses (ie which take longer than 1 second) - CONFIG_CIFS_STATS2 must be enabled in .config 0x04 - - -traceSMB If set to one, debug information is logged to the - system error log with the start of smb requests - and responses (default 0) -LookupCacheEnable If set to one, inode information is kept cached - for one second improving performance of lookups - (default 1) -LinuxExtensionsEnabled If set to one then the client will attempt to - use the CIFS "UNIX" extensions which are optional - protocol enhancements that allow CIFS servers - to return accurate UID/GID information as well - as support symbolic links. If you use servers - such as Samba that support the CIFS Unix - extensions but do not want to use symbolic link - support and want to map the uid and gid fields - to values supplied at mount (rather than the - actual values, then set this to zero. (default 1) - -These experimental features and tracing can be enabled by changing flags in -/proc/fs/cifs (after the cifs module has been installed or built into the -kernel, e.g. insmod cifs). To enable a feature set it to 1 e.g. to enable -tracing to the kernel message log type: - - echo 7 > /proc/fs/cifs/cifsFYI - -cifsFYI functions as a bit mask. Setting it to 1 enables additional kernel -logging of various informational messages. 2 enables logging of non-zero -SMB return codes while 4 enables logging of requests that take longer -than one second to complete (except for byte range lock requests). -Setting it to 4 requires CONFIG_CIFS_STATS2 to be set in kernel configuration -(.config). Setting it to seven enables all three. Finally, tracing -the start of smb requests and responses can be enabled via: - - echo 1 > /proc/fs/cifs/traceSMB - -Per share (per client mount) statistics are available in /proc/fs/cifs/Stats. -Additional information is available if CONFIG_CIFS_STATS2 is enabled in the -kernel configuration (.config). The statistics returned include counters which -represent the number of attempted and failed (ie non-zero return code from the -server) SMB3 (or cifs) requests grouped by request type (read, write, close etc.). -Also recorded is the total bytes read and bytes written to the server for -that share. Note that due to client caching effects this can be less than the -number of bytes read and written by the application running on the client. -Statistics can be reset to zero by "echo 0 > /proc/fs/cifs/Stats" which may be -useful if comparing performance of two different scenarios. - -Also note that "cat /proc/fs/cifs/DebugData" will display information about -the active sessions and the shares that are mounted. - -Enabling Kerberos (extended security) works but requires version 1.2 or later -of the helper program cifs.upcall to be present and to be configured in the -/etc/request-key.conf file. The cifs.upcall helper program is from the Samba -project(http://www.samba.org). NTLM and NTLMv2 and LANMAN support do not -require this helper. Note that NTLMv2 security (which does not require the -cifs.upcall helper program), instead of using Kerberos, is sufficient for -some use cases. - -DFS support allows transparent redirection to shares in an MS-DFS name space. -In addition, DFS support for target shares which are specified as UNC -names which begin with host names (rather than IP addresses) requires -a user space helper (such as cifs.upcall) to be present in order to -translate host names to ip address, and the user space helper must also -be configured in the file /etc/request-key.conf. Samba, Windows servers and -many NAS appliances support DFS as a way of constructing a global name -space to ease network configuration and improve reliability. - -To use cifs Kerberos and DFS support, the Linux keyutils package should be -installed and something like the following lines should be added to the -/etc/request-key.conf file: - -create cifs.spnego * * /usr/local/sbin/cifs.upcall %k -create dns_resolver * * /usr/local/sbin/cifs.upcall %k - -CIFS kernel module parameters -============================= -These module parameters can be specified or modified either during the time of -module loading or during the runtime by using the interface - /proc/module/cifs/parameters/<param> - -i.e. echo "value" > /sys/module/cifs/parameters/<param> - -1. enable_oplocks - Enable or disable oplocks. Oplocks are enabled by default. - [Y/y/1]. To disable use any of [N/n/0]. - diff --git a/Documentation/filesystems/cifs/TODO b/Documentation/filesystems/cifs/TODO deleted file mode 100644 index 9267f3fb131f..000000000000 --- a/Documentation/filesystems/cifs/TODO +++ /dev/null @@ -1,119 +0,0 @@ -Version 2.14 December 21, 2018 - -A Partial List of Missing Features -================================== - -Contributions are welcome. There are plenty of opportunities -for visible, important contributions to this module. Here -is a partial list of the known problems and missing features: - -a) SMB3 (and SMB3.1.1) missing optional features: - - multichannel (started), integration with RDMA - - directory leases (improved metadata caching), started (root dir only) - - T10 copy offload ie "ODX" (copy chunk, and "Duplicate Extents" ioctl - currently the only two server side copy mechanisms supported) - -b) improved sparse file support - -c) Directory entry caching relies on a 1 second timer, rather than -using Directory Leases, currently only the root file handle is cached longer - -d) quota support (needs minor kernel change since quota calls -to make it to network filesystems or deviceless filesystems) - -e) Additional use cases where we use "compoounding" (e.g. open/query/close -and open/setinfo/close) to reduce the number of roundtrips, and also -open to reduce redundant opens (using deferred close and reference counts more). - -f) Finish inotify support so kde and gnome file list windows -will autorefresh (partially complete by Asser). Needs minor kernel -vfs change to support removing D_NOTIFY on a file. - -g) Add GUI tool to configure /proc/fs/cifs settings and for display of -the CIFS statistics (started) - -h) implement support for security and trusted categories of xattrs -(requires minor protocol extension) to enable better support for SELINUX - -i) Add support for tree connect contexts (see MS-SMB2) a new SMB3.1.1 protocol - feature (may be especially useful for virtualization). - -j) Create UID mapping facility so server UIDs can be mapped on a per -mount or a per server basis to client UIDs or nobody if no mapping -exists. Also better integration with winbind for resolving SID owners - -k) Add tools to take advantage of more smb3 specific ioctls and features -(passthrough ioctl/fsctl for sending various SMB3 fsctls to the server -is in progress, and a passthrough query_info call is already implemented -in cifs.ko to allow smb3 info levels queries to be sent from userspace) - -l) encrypted file support - -m) improved stats gathering tools (perhaps integration with nfsometer?) -to extend and make easier to use what is currently in /proc/fs/cifs/Stats - -n) allow setting more NTFS/SMB3 file attributes remotely (currently limited to compressed -file attribute via chflags) and improve user space tools for managing and -viewing them. - -o) mount helper GUI (to simplify the various configuration options on mount) - -p) Add support for witness protocol (perhaps ioctl to cifs.ko from user space - tool listening on witness protocol RPC) to allow for notification of share - move, server failover, and server adapter changes. And also improve other - failover scenarios, e.g. when client knows multiple DFS entries point to - different servers, and the server we are connected to has gone down. - -q) Allow mount.cifs to be more verbose in reporting errors with dialect -or unsupported feature errors. - -r) updating cifs documentation, and user guide. - -s) Addressing bugs found by running a broader set of xfstests in standard -file system xfstest suite. - -t) split cifs and smb3 support into separate modules so legacy (and less -secure) CIFS dialect can be disabled in environments that don't need it -and simplify the code. - -v) POSIX Extensions for SMB3.1.1 (started, create and mkdir support added -so far). - -w) Add support for additional strong encryption types, and additional spnego -authentication mechanisms (see MS-SMB2) - -KNOWN BUGS -==================================== -See http://bugzilla.samba.org - search on product "CifsVFS" for -current bug list. Also check http://bugzilla.kernel.org (Product = File System, Component = CIFS) - -1) existing symbolic links (Windows reparse points) are recognized but -can not be created remotely. They are implemented for Samba and those that -support the CIFS Unix extensions, although earlier versions of Samba -overly restrict the pathnames. -2) follow_link and readdir code does not follow dfs junctions -but recognizes them - -Misc testing to do -================== -1) check out max path names and max path name components against various server -types. Try nested symlinks (8 deep). Return max path name in stat -f information - -2) Improve xfstest's cifs/smb3 enablement and adapt xfstests where needed to test -cifs/smb3 better - -3) Additional performance testing and optimization using iozone and similar - -there are some easy changes that can be done to parallelize sequential writes, -and when signing is disabled to request larger read sizes (larger than -negotiated size) and send larger write sizes to modern servers. - -4) More exhaustively test against less common servers - -5) Continue to extend the smb3 "buildbot" which does automated xfstesting -against Windows, Samba and Azure currently - to add additional tests and -to allow the buildbot to execute the tests faster. The URL for the -buildbot is: http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com - -6) Address various coverity warnings (most are not bugs per-se, but -the more warnings are addressed, the easier it is to spot real -problems that static analyzers will point out in the future). diff --git a/Documentation/filesystems/cifs/cifs.txt b/Documentation/filesystems/cifs/cifs.txt deleted file mode 100644 index 1be3d21c286e..000000000000 --- a/Documentation/filesystems/cifs/cifs.txt +++ /dev/null @@ -1,45 +0,0 @@ - This is the client VFS module for the SMB3 NAS protocol as well - as for older dialects such as the Common Internet File System (CIFS) - protocol which was the successor to the Server Message Block - (SMB) protocol, the native file sharing mechanism for most early - PC operating systems. New and improved versions of CIFS are now - called SMB2 and SMB3. Use of SMB3 (and later, including SMB3.1.1) - is strongly preferred over using older dialects like CIFS due to - security reaasons. All modern dialects, including the most recent, - SMB3.1.1 are supported by the CIFS VFS module. The SMB3 protocol - is implemented and supported by all major file servers - such as all modern versions of Windows (including Windows 2016 - Server), as well as by Samba (which provides excellent - CIFS/SMB2/SMB3 server support and tools for Linux and many other - operating systems). Apple systems also support SMB3 well, as - do most Network Attached Storage vendors, so this network - filesystem client can mount to a wide variety of systems. - It also supports mounting to the cloud (for example - Microsoft Azure), including the necessary security features. - - The intent of this module is to provide the most advanced network - file system function for SMB3 compliant servers, including advanced - security features, excellent parallelized high performance i/o, better - POSIX compliance, secure per-user session establishment, encryption, - high performance safe distributed caching (leases/oplocks), optional packet - signing, large files, Unicode support and other internationalization - improvements. Since both Samba server and this filesystem client support - the CIFS Unix extensions (and in the future SMB3 POSIX extensions), - the combination can provide a reasonable alternative to other network and - cluster file systems for fileserving in some Linux to Linux environments, - not just in Linux to Windows (or Linux to Mac) environments. - - This filesystem has a mount utility (mount.cifs) and various user space - tools (including smbinfo and setcifsacl) that can be obtained from - - https://git.samba.org/?p=cifs-utils.git - or - git://git.samba.org/cifs-utils.git - - mount.cifs should be installed in the directory with the other mount helpers. - - For more information on the module see the project wiki page at - - https://wiki.samba.org/index.php/LinuxCIFS - and - https://wiki.samba.org/index.php/LinuxCIFS_utils diff --git a/Documentation/filesystems/cifs/cifsroot.txt b/Documentation/filesystems/cifs/cifsroot.txt new file mode 100644 index 000000000000..0fa1a2c36a40 --- /dev/null +++ b/Documentation/filesystems/cifs/cifsroot.txt @@ -0,0 +1,97 @@ +Mounting root file system via SMB (cifs.ko) +=========================================== + +Written 2019 by Paulo Alcantara <palcantara@suse.de> +Written 2019 by Aurelien Aptel <aaptel@suse.com> + +The CONFIG_CIFS_ROOT option enables experimental root file system +support over the SMB protocol via cifs.ko. + +It introduces a new kernel command-line option called 'cifsroot=' +which will tell the kernel to mount the root file system over the +network by utilizing SMB or CIFS protocol. + +In order to mount, the network stack will also need to be set up by +using 'ip=' config option. For more details, see +Documentation/filesystems/nfs/nfsroot.txt. + +A CIFS root mount currently requires the use of SMB1+UNIX Extensions +which is only supported by the Samba server. SMB1 is the older +deprecated version of the protocol but it has been extended to support +POSIX features (See [1]). The equivalent extensions for the newer +recommended version of the protocol (SMB3) have not been fully +implemented yet which means SMB3 doesn't support some required POSIX +file system objects (e.g. block devices, pipes, sockets). + +As a result, a CIFS root will default to SMB1 for now but the version +to use can nonetheless be changed via the 'vers=' mount option. This +default will change once the SMB3 POSIX extensions are fully +implemented. + +Server configuration +==================== + +To enable SMB1+UNIX extensions you will need to set these global +settings in Samba smb.conf: + + [global] + server min protocol = NT1 + unix extension = yes # default + +Kernel command line +=================== + +root=/dev/cifs + +This is just a virtual device that basically tells the kernel to mount +the root file system via SMB protocol. + +cifsroot=//<server-ip>/<share>[,options] + +Enables the kernel to mount the root file system via SMB that are +located in the <server-ip> and <share> specified in this option. + +The default mount options are set in fs/cifs/cifsroot.c. + +server-ip + IPv4 address of the server. + +share + Path to SMB share (rootfs). + +options + Optional mount options. For more information, see mount.cifs(8). + +Examples +======== + +Export root file system as a Samba share in smb.conf file. + +... +[linux] + path = /path/to/rootfs + read only = no + guest ok = yes + force user = root + force group = root + browseable = yes + writeable = yes + admin users = root + public = yes + create mask = 0777 + directory mask = 0777 +... + +Restart smb service. + +# systemctl restart smb + +Test it under QEMU on a kernel built with CONFIG_CIFS_ROOT and +CONFIG_IP_PNP options enabled. + +# qemu-system-x86_64 -enable-kvm -cpu host -m 1024 \ + -kernel /path/to/linux/arch/x86/boot/bzImage -nographic \ + -append "root=/dev/cifs rw ip=dhcp cifsroot=//10.0.2.2/linux,username=foo,password=bar console=ttyS0 3" + + +1: https://wiki.samba.org/index.php/UNIX_Extensions diff --git a/Documentation/filesystems/cifs/winucase_convert.pl b/Documentation/filesystems/cifs/winucase_convert.pl deleted file mode 100755 index 322a9c833f23..000000000000 --- a/Documentation/filesystems/cifs/winucase_convert.pl +++ /dev/null @@ -1,62 +0,0 @@ -#!/usr/bin/perl -w -# -# winucase_convert.pl -- convert "Windows 8 Upper Case Mapping Table.txt" to -# a two-level set of C arrays. -# -# Copyright 2013: Jeff Layton <jlayton@redhat.com> -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# - -while(<>) { - next if (!/^0x(..)(..)\t0x(....)\t/); - $firstchar = hex($1); - $secondchar = hex($2); - $uppercase = hex($3); - - $top[$firstchar][$secondchar] = $uppercase; -} - -for ($i = 0; $i < 256; $i++) { - next if (!$top[$i]); - - printf("static const wchar_t t2_%2.2x[256] = {", $i); - for ($j = 0; $j < 256; $j++) { - if (($j % 8) == 0) { - print "\n\t"; - } else { - print " "; - } - printf("0x%4.4x,", $top[$i][$j] ? $top[$i][$j] : 0); - } - print "\n};\n\n"; -} - -printf("static const wchar_t *const toplevel[256] = {", $i); -for ($i = 0; $i < 256; $i++) { - if (($i % 8) == 0) { - print "\n\t"; - } elsif ($top[$i]) { - print " "; - } else { - print " "; - } - - if ($top[$i]) { - printf("t2_%2.2x,", $i); - } else { - print "NULL,"; - } -} -print "\n};\n\n"; diff --git a/Documentation/filesystems/coda.txt b/Documentation/filesystems/coda.txt index 61311356025d..1711ad48e38a 100644 --- a/Documentation/filesystems/coda.txt +++ b/Documentation/filesystems/coda.txt @@ -421,14 +421,14 @@ kernel support. The CodaCred structure defines a variety of user and group ids as - they are set for the calling process. The vuid_t and guid_t are 32 bit + they are set for the calling process. The vuid_t and vgid_t are 32 bit unsigned integers. It also defines group membership in an array. On Unix the CodaCred has proven sufficient to implement good security semantics for Coda but the structure may have to undergo modification for the Windows environment when these mature. struct CodaCred { - vuid_t cr_uid, cr_euid, cr_suid, cr_fsuid; /* Real, effective, set, fs uid*/ + vuid_t cr_uid, cr_euid, cr_suid, cr_fsuid; /* Real, effective, set, fs uid */ vgid_t cr_gid, cr_egid, cr_sgid, cr_fsgid; /* same for groups */ vgid_t cr_groups[NGROUPS]; /* Group membership for caller */ }; @@ -481,7 +481,10 @@ kernel support. - + struct coda_timespec { + int64_t tv_sec; /* seconds */ + long tv_nsec; /* nanoseconds */ + }; struct coda_vattr { enum coda_vtype va_type; /* vnode type (for create) */ @@ -493,9 +496,9 @@ kernel support. long va_fileid; /* file id */ u_quad_t va_size; /* file size in bytes */ long va_blocksize; /* blocksize preferred for i/o */ - struct timespec va_atime; /* time of last access */ - struct timespec va_mtime; /* time of last modification */ - struct timespec va_ctime; /* time file changed */ + struct coda_timespec va_atime; /* time of last access */ + struct coda_timespec va_mtime; /* time of last modification */ + struct coda_timespec va_ctime; /* time file changed */ u_long va_gen; /* generation number of file */ u_long va_flags; /* flags defined for file */ dev_t va_rdev; /* device special file represents */ diff --git a/Documentation/filesystems/conf.py b/Documentation/filesystems/conf.py deleted file mode 100644 index ea44172af5c4..000000000000 --- a/Documentation/filesystems/conf.py +++ /dev/null @@ -1,10 +0,0 @@ -# -*- coding: utf-8; mode: python -*- - -project = "Linux Filesystems API" - -tags.add("subproject") - -latex_documents = [ - ('index', 'filesystems.tex', project, - 'The kernel development community', 'manual'), -] diff --git a/Documentation/filesystems/dax.txt b/Documentation/filesystems/dax.txt index 6d2c0d340dea..679729442fd2 100644 --- a/Documentation/filesystems/dax.txt +++ b/Documentation/filesystems/dax.txt @@ -76,7 +76,7 @@ exposure of uninitialized data through mmap. These filesystems may be used for inspiration: - ext2: see Documentation/filesystems/ext2.txt - ext4: see Documentation/filesystems/ext4/ -- xfs: see Documentation/filesystems/xfs.txt +- xfs: see Documentation/admin-guide/xfs.rst Handling Media Errors diff --git a/Documentation/filesystems/debugfs.txt b/Documentation/filesystems/debugfs.txt index 4a0a9c3f4af6..9e27c843d00e 100644 --- a/Documentation/filesystems/debugfs.txt +++ b/Documentation/filesystems/debugfs.txt @@ -169,7 +169,7 @@ byte offsets over a base for the register block. If you want to dump an u32 array in debugfs, you can create file with: - struct dentry *debugfs_create_u32_array(const char *name, umode_t mode, + void debugfs_create_u32_array(const char *name, umode_t mode, struct dentry *parent, u32 *array, u32 elements); diff --git a/Documentation/filesystems/directory-locking b/Documentation/filesystems/directory-locking.rst index 4e32cb961e5b..de12016ee419 100644 --- a/Documentation/filesystems/directory-locking +++ b/Documentation/filesystems/directory-locking.rst @@ -1,12 +1,17 @@ - Locking scheme used for directory operations is based on two +================= +Directory Locking +================= + + +Locking scheme used for directory operations is based on two kinds of locks - per-inode (->i_rwsem) and per-filesystem (->s_vfs_rename_mutex). - When taking the i_rwsem on multiple non-directory objects, we +When taking the i_rwsem on multiple non-directory objects, we always acquire the locks in order by increasing address. We'll call that "inode pointer" order in the following. - For our purposes all operations fall in 5 classes: +For our purposes all operations fall in 5 classes: 1) read access. Locking rules: caller locks directory we are accessing. The lock is taken shared. @@ -27,25 +32,29 @@ NB: we might get away with locking the the source (and target in exchange case) shared. 5) link creation. Locking rules: + * lock parent * check that source is not a directory * lock source * call the method. + All locks are exclusive. 6) cross-directory rename. The trickiest in the whole bunch. Locking rules: + * lock the filesystem * lock parents in "ancestors first" order. * find source and target. * if old parent is equal to or is a descendent of target - fail with -ENOTEMPTY + fail with -ENOTEMPTY * if new parent is equal to or is a descendent of source - fail with -ELOOP + fail with -ELOOP * If it's an exchange, lock both the source and the target. * If the target exists, lock it. If the source is a non-directory, lock it. If we need to lock both, do so in inode pointer order. * call the method. + All ->i_rwsem are taken exclusive. Again, we might get away with locking the the source (and target in exchange case) shared. @@ -54,10 +63,11 @@ read, modified or removed by method will be locked by caller. If no directory is its own ancestor, the scheme above is deadlock-free. + Proof: First of all, at any moment we have a partial ordering of the -objects - A < B iff A is an ancestor of B. + objects - A < B iff A is an ancestor of B. That ordering can change. However, the following is true: @@ -77,32 +87,32 @@ objects - A < B iff A is an ancestor of B. non-directory object, except renames, which take locks on source and target in inode pointer order in the case they are not directories.) - Now consider the minimal deadlock. Each process is blocked on +Now consider the minimal deadlock. Each process is blocked on attempt to acquire some lock and already holds at least one lock. Let's consider the set of contended locks. First of all, filesystem lock is not contended, since any process blocked on it is not holding any locks. Thus all processes are blocked on ->i_rwsem. - By (3), any process holding a non-directory lock can only be +By (3), any process holding a non-directory lock can only be waiting on another non-directory lock with a larger address. Therefore the process holding the "largest" such lock can always make progress, and non-directory objects are not included in the set of contended locks. - Thus link creation can't be a part of deadlock - it can't be +Thus link creation can't be a part of deadlock - it can't be blocked on source and it means that it doesn't hold any locks. - Any contended object is either held by cross-directory rename or +Any contended object is either held by cross-directory rename or has a child that is also contended. Indeed, suppose that it is held by operation other than cross-directory rename. Then the lock this operation is blocked on belongs to child of that object due to (1). - It means that one of the operations is cross-directory rename. +It means that one of the operations is cross-directory rename. Otherwise the set of contended objects would be infinite - each of them would have a contended child and we had assumed that no object is its own descendent. Moreover, there is exactly one cross-directory rename (see above). - Consider the object blocking the cross-directory rename. One +Consider the object blocking the cross-directory rename. One of its descendents is locked by cross-directory rename (otherwise we would again have an infinite set of contended objects). But that means that cross-directory rename is taking locks out of order. Due @@ -112,7 +122,7 @@ try to acquire lock on descendent before the lock on ancestor. Contradiction. I.e. deadlock is impossible. Q.E.D. - These operations are guaranteed to avoid loop creation. Indeed, +These operations are guaranteed to avoid loop creation. Indeed, the only operation that could introduce loops is cross-directory rename. Since the only new (parent, child) pair added by rename() is (new parent, source), such loop would have to contain these objects and the rest of it @@ -123,13 +133,13 @@ new parent had been equal to or a descendent of source since the moment when we had acquired filesystem lock and rename() would fail with -ELOOP in that case. - While this locking scheme works for arbitrary DAGs, it relies on +While this locking scheme works for arbitrary DAGs, it relies on ability to check that directory is a descendent of another object. Current implementation assumes that directory graph is a tree. This assumption is also preserved by all operations (cross-directory rename on a tree that would not introduce a cycle will leave it a tree and link() fails for directories). - Notice that "directory" in the above == "anything that might have +Notice that "directory" in the above == "anything that might have children", so if we are going to introduce hybrid objects we will need either to make sure that link(2) doesn't work for them or to make changes in is_subdir() that would make it work even in presence of such beasts. diff --git a/Documentation/filesystems/erofs.txt b/Documentation/filesystems/erofs.txt new file mode 100644 index 000000000000..b0c085326e2e --- /dev/null +++ b/Documentation/filesystems/erofs.txt @@ -0,0 +1,210 @@ +Overview +======== + +EROFS file-system stands for Enhanced Read-Only File System. Different +from other read-only file systems, it aims to be designed for flexibility, +scalability, but be kept simple and high performance. + +It is designed as a better filesystem solution for the following scenarios: + - read-only storage media or + + - part of a fully trusted read-only solution, which means it needs to be + immutable and bit-for-bit identical to the official golden image for + their releases due to security and other considerations and + + - hope to save some extra storage space with guaranteed end-to-end performance + by using reduced metadata and transparent file compression, especially + for those embedded devices with limited memory (ex, smartphone); + +Here is the main features of EROFS: + - Little endian on-disk design; + + - Currently 4KB block size (nobh) and therefore maximum 16TB address space; + + - Metadata & data could be mixed by design; + + - 2 inode versions for different requirements: + v1 v2 + Inode metadata size: 32 bytes 64 bytes + Max file size: 4 GB 16 EB (also limited by max. vol size) + Max uids/gids: 65536 4294967296 + File creation time: no yes (64 + 32-bit timestamp) + Max hardlinks: 65536 4294967296 + Metadata reserved: 4 bytes 14 bytes + + - Support extended attributes (xattrs) as an option; + + - Support xattr inline and tail-end data inline for all files; + + - Support POSIX.1e ACLs by using xattrs; + + - Support transparent file compression as an option: + LZ4 algorithm with 4 KB fixed-output compression for high performance; + +The following git tree provides the file system user-space tools under +development (ex, formatting tool mkfs.erofs): +>> git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git + +Bugs and patches are welcome, please kindly help us and send to the following +linux-erofs mailing list: +>> linux-erofs mailing list <linux-erofs@lists.ozlabs.org> + +Mount options +============= + +(no)user_xattr Setup Extended User Attributes. Note: xattr is enabled + by default if CONFIG_EROFS_FS_XATTR is selected. +(no)acl Setup POSIX Access Control List. Note: acl is enabled + by default if CONFIG_EROFS_FS_POSIX_ACL is selected. +cache_strategy=%s Select a strategy for cached decompression from now on: + disabled: In-place I/O decompression only; + readahead: Cache the last incomplete compressed physical + cluster for further reading. It still does + in-place I/O decompression for the rest + compressed physical clusters; + readaround: Cache the both ends of incomplete compressed + physical clusters for further reading. + It still does in-place I/O decompression + for the rest compressed physical clusters. + +On-disk details +=============== + +Summary +------- +Different from other read-only file systems, an EROFS volume is designed +to be as simple as possible: + + |-> aligned with the block size + ____________________________________________________________ + | |SB| | ... | Metadata | ... | Data | Metadata | ... | Data | + |_|__|_|_____|__________|_____|______|__________|_____|______| + 0 +1K + +All data areas should be aligned with the block size, but metadata areas +may not. All metadatas can be now observed in two different spaces (views): + 1. Inode metadata space + Each valid inode should be aligned with an inode slot, which is a fixed + value (32 bytes) and designed to be kept in line with v1 inode size. + + Each inode can be directly found with the following formula: + inode offset = meta_blkaddr * block_size + 32 * nid + + |-> aligned with 8B + |-> followed closely + + meta_blkaddr blocks |-> another slot + _____________________________________________________________________ + | ... | inode | xattrs | extents | data inline | ... | inode ... + |________|_______|(optional)|(optional)|__(optional)_|_____|__________ + |-> aligned with the inode slot size + . . + . . + . . + . . + . . + . . + .____________________________________________________|-> aligned with 4B + | xattr_ibody_header | shared xattrs | inline xattrs | + |____________________|_______________|_______________| + |-> 12 bytes <-|->x * 4 bytes<-| . + . . . + . . . + . . . + ._______________________________.______________________. + | id | id | id | id | ... | id | ent | ... | ent| ... | + |____|____|____|____|______|____|_____|_____|____|_____| + |-> aligned with 4B + |-> aligned with 4B + + Inode could be 32 or 64 bytes, which can be distinguished from a common + field which all inode versions have -- i_advise: + + __________________ __________________ + | i_advise | | i_advise | + |__________________| |__________________| + | ... | | ... | + | | | | + |__________________| 32 bytes | | + | | + |__________________| 64 bytes + + Xattrs, extents, data inline are followed by the corresponding inode with + proper alignes, and they could be optional for different data mappings, + _currently_ there are totally 3 valid data mappings supported: + + 1) flat file data without data inline (no extent); + 2) fixed-output size data compression (must have extents); + 3) flat file data with tail-end data inline (no extent); + + The size of the optional xattrs is indicated by i_xattr_count in inode + header. Large xattrs or xattrs shared by many different files can be + stored in shared xattrs metadata rather than inlined right after inode. + + 2. Shared xattrs metadata space + Shared xattrs space is similar to the above inode space, started with + a specific block indicated by xattr_blkaddr, organized one by one with + proper align. + + Each share xattr can also be directly found by the following formula: + xattr offset = xattr_blkaddr * block_size + 4 * xattr_id + + |-> aligned by 4 bytes + + xattr_blkaddr blocks |-> aligned with 4 bytes + _________________________________________________________________________ + | ... | xattr_entry | xattr data | ... | xattr_entry | xattr data ... + |________|_____________|_____________|_____|______________|_______________ + +Directories +----------- +All directories are now organized in a compact on-disk format. Note that +each directory block is divided into index and name areas in order to support +random file lookup, and all directory entries are _strictly_ recorded in +alphabetical order in order to support improved prefix binary search +algorithm (could refer to the related source code). + + ___________________________ + / | + / ______________|________________ + / / | nameoff1 | nameoffN-1 + ____________.______________._______________v________________v__________ +| dirent | dirent | ... | dirent | filename | filename | ... | filename | +|___.0___|____1___|_____|___N-1__|____0_____|____1_____|_____|___N-1____| + \ ^ + \ | * could have + \ | trailing '\0' + \________________________| nameoff0 + + Directory block + +Note that apart from the offset of the first filename, nameoff0 also indicates +the total number of directory entries in this block since it is no need to +introduce another on-disk field at all. + +Compression +----------- +Currently, EROFS supports 4KB fixed-output clustersize transparent file +compression, as illustrated below: + + |---- Variant-Length Extent ----|-------- VLE --------|----- VLE ----- + clusterofs clusterofs clusterofs + | | | logical data +_________v_______________________________v_____________________v_______________ +... | . | | . | | . | ... +____|____.________|_____________|________.____|_____________|__.__________|____ + |-> cluster <-|-> cluster <-|-> cluster <-|-> cluster <-|-> cluster <-| + size size size size size + . . . . + . . . . + . . . . + _______._____________._____________._____________._____________________ + ... | | | | ... physical data + _______|_____________|_____________|_____________|_____________________ + |-> cluster <-|-> cluster <-|-> cluster <-| + size size size + +Currently each on-disk physical cluster can contain 4KB (un)compressed data +at most. For each logical cluster, there is a corresponding on-disk index to +describe its cluster type, physical cluster address, etc. + +See "struct z_erofs_vle_decompressed_index" in erofs_fs.h for more details. + diff --git a/Documentation/filesystems/ext2.txt b/Documentation/filesystems/ext2.txt index a19973a4dd1e..94c2cf0292f5 100644 --- a/Documentation/filesystems/ext2.txt +++ b/Documentation/filesystems/ext2.txt @@ -57,7 +57,13 @@ noacl Don't support POSIX ACLs. nobh Do not attach buffer_heads to file pagecache. -grpquota,noquota,quota,usrquota Quota options are silently ignored by ext2. +quota, usrquota Enable user disk quota support + (requires CONFIG_QUOTA). + +grpquota Enable group disk quota support + (requires CONFIG_QUOTA). + +noquota option ls silently ignored by ext2. Specification diff --git a/Documentation/filesystems/ext4/bigalloc.rst b/Documentation/filesystems/ext4/bigalloc.rst index c6d88557553c..72075aa608e4 100644 --- a/Documentation/filesystems/ext4/bigalloc.rst +++ b/Documentation/filesystems/ext4/bigalloc.rst @@ -9,14 +9,26 @@ ext4 code is not prepared to handle the case where the block size exceeds the page size. However, for a filesystem of mostly huge files, it is desirable to be able to allocate disk blocks in units of multiple blocks to reduce both fragmentation and metadata overhead. The -`bigalloc <Bigalloc>`__ feature provides exactly this ability. The -administrator can set a block cluster size at mkfs time (which is stored -in the s\_log\_cluster\_size field in the superblock); from then on, the -block bitmaps track clusters, not individual blocks. This means that -block groups can be several gigabytes in size (instead of just 128MiB); -however, the minimum allocation unit becomes a cluster, not a block, -even for directories. TaoBao had a patchset to extend the “use units of -clusters instead of blocks” to the extent tree, though it is not clear -where those patches went-- they eventually morphed into “extent tree v2” -but that code has not landed as of May 2015. +bigalloc feature provides exactly this ability. + +The bigalloc feature (EXT4_FEATURE_RO_COMPAT_BIGALLOC) changes ext4 to +use clustered allocation, so that each bit in the ext4 block allocation +bitmap addresses a power of two number of blocks. For example, if the +file system is mainly going to be storing large files in the 4-32 +megabyte range, it might make sense to set a cluster size of 1 megabyte. +This means that each bit in the block allocation bitmap now addresses +256 4k blocks. This shrinks the total size of the block allocation +bitmaps for a 2T file system from 64 megabytes to 256 kilobytes. It also +means that a block group addresses 32 gigabytes instead of 128 megabytes, +also shrinking the amount of file system overhead for metadata. + +The administrator can set a block cluster size at mkfs time (which is +stored in the s\_log\_cluster\_size field in the superblock); from then +on, the block bitmaps track clusters, not individual blocks. This means +that block groups can be several gigabytes in size (instead of just +128MiB); however, the minimum allocation unit becomes a cluster, not a +block, even for directories. TaoBao had a patchset to extend the “use +units of clusters instead of blocks” to the extent tree, though it is +not clear where those patches went-- they eventually morphed into +“extent tree v2” but that code has not landed as of May 2015. diff --git a/Documentation/filesystems/ext4/blockgroup.rst b/Documentation/filesystems/ext4/blockgroup.rst index baf888e4c06a..3da156633339 100644 --- a/Documentation/filesystems/ext4/blockgroup.rst +++ b/Documentation/filesystems/ext4/blockgroup.rst @@ -71,11 +71,11 @@ if the flex\_bg size is 4, then group 0 will contain (in order) the superblock, group descriptors, data block bitmaps for groups 0-3, inode bitmaps for groups 0-3, inode tables for groups 0-3, and the remaining space in group 0 is for file data. The effect of this is to group the -block metadata close together for faster loading, and to enable large -files to be continuous on disk. Backup copies of the superblock and -group descriptors are always at the beginning of block groups, even if -flex\_bg is enabled. The number of block groups that make up a flex\_bg -is given by 2 ^ ``sb.s_log_groups_per_flex``. +block group metadata close together for faster loading, and to enable +large files to be continuous on disk. Backup copies of the superblock +and group descriptors are always at the beginning of block groups, even +if flex\_bg is enabled. The number of block groups that make up a +flex\_bg is given by 2 ^ ``sb.s_log_groups_per_flex``. Meta Block Groups ----------------- diff --git a/Documentation/filesystems/ext4/blocks.rst b/Documentation/filesystems/ext4/blocks.rst index 73d4dc0f7bda..bd722ecd92d6 100644 --- a/Documentation/filesystems/ext4/blocks.rst +++ b/Documentation/filesystems/ext4/blocks.rst @@ -10,7 +10,9 @@ block groups. Block size is specified at mkfs time and typically is 4KiB. You may experience mounting problems if block size is greater than page size (i.e. 64KiB blocks on a i386 which only has 4KiB memory pages). By default a filesystem can contain 2^32 blocks; if the '64bit' -feature is enabled, then a filesystem can have 2^64 blocks. +feature is enabled, then a filesystem can have 2^64 blocks. The location +of structures is stored in terms of the block number the structure lives +in and not the absolute offset on disk. For 32-bit filesystems, limits are as follows: diff --git a/Documentation/filesystems/ext4/directory.rst b/Documentation/filesystems/ext4/directory.rst index 614034e24669..073940cc64ed 100644 --- a/Documentation/filesystems/ext4/directory.rst +++ b/Documentation/filesystems/ext4/directory.rst @@ -59,7 +59,7 @@ is at most 263 bytes long, though on disk you'll need to reference - File name. Since file names cannot be longer than 255 bytes, the new directory -entry format shortens the rec\_len field and uses the space for a file +entry format shortens the name\_len field and uses the space for a file type flag, probably to avoid having to load every inode during directory tree traversal. This format is ``ext4_dir_entry_2``, which is at most 263 bytes long, though on disk you'll need to reference diff --git a/Documentation/filesystems/ext4/group_descr.rst b/Documentation/filesystems/ext4/group_descr.rst index 0f783ed88592..7ba6114e7f5c 100644 --- a/Documentation/filesystems/ext4/group_descr.rst +++ b/Documentation/filesystems/ext4/group_descr.rst @@ -99,9 +99,12 @@ The block group descriptor is laid out in ``struct ext4_group_desc``. * - 0x1E - \_\_le16 - bg\_checksum - - Group descriptor checksum; crc16(sb\_uuid+group+desc) if the - RO\_COMPAT\_GDT\_CSUM feature is set, or crc32c(sb\_uuid+group\_desc) & - 0xFFFF if the RO\_COMPAT\_METADATA\_CSUM feature is set. + - Group descriptor checksum; crc16(sb\_uuid+group\_num+bg\_desc) if the + RO\_COMPAT\_GDT\_CSUM feature is set, or + crc32c(sb\_uuid+group\_num+bg\_desc) & 0xFFFF if the + RO\_COMPAT\_METADATA\_CSUM feature is set. The bg\_checksum + field in bg\_desc is skipped when calculating crc16 checksum, + and set to zero if crc32c checksum is used. * - - - diff --git a/Documentation/filesystems/ext4/index.rst b/Documentation/filesystems/ext4/index.rst index 3be3e54d480d..705d813d558f 100644 --- a/Documentation/filesystems/ext4/index.rst +++ b/Documentation/filesystems/ext4/index.rst @@ -8,7 +8,7 @@ ext4 Data Structures and Algorithms :maxdepth: 6 :numbered: - about.rst - overview.rst - globals.rst - dynamic.rst + about + overview + globals + dynamic diff --git a/Documentation/filesystems/ext4/inodes.rst b/Documentation/filesystems/ext4/inodes.rst index 6bd35e506b6f..a65baffb4ebf 100644 --- a/Documentation/filesystems/ext4/inodes.rst +++ b/Documentation/filesystems/ext4/inodes.rst @@ -277,6 +277,8 @@ The ``i_flags`` field is a combination of these values: - This is a huge file (EXT4\_HUGE\_FILE\_FL). * - 0x80000 - Inode uses extents (EXT4\_EXTENTS\_FL). + * - 0x100000 + - Verity protected file (EXT4\_VERITY\_FL). * - 0x200000 - Inode stores a large extended attribute value in its data blocks (EXT4\_EA\_INODE\_FL). @@ -299,9 +301,9 @@ The ``i_flags`` field is a combination of these values: - Reserved for ext4 library (EXT4\_RESERVED\_FL). * - - Aggregate flags: - * - 0x4BDFFF + * - 0x705BDFFF - User-visible flags. - * - 0x4B80FF + * - 0x604BC0FF - User-modifiable flags. Note that while EXT4\_JOURNAL\_DATA\_FL and EXT4\_EXTENTS\_FL can be set with setattr, they are not in the kernel's EXT4\_FL\_USER\_MODIFIABLE mask, since it needs to handle the setting of @@ -470,8 +472,8 @@ inode, which allows struct ext4\_inode to grow for a new kernel without having to upgrade all of the on-disk inodes. Access to fields beyond EXT2\_GOOD\_OLD\_INODE\_SIZE should be verified to be within ``i_extra_isize``. By default, ext4 inode records are 256 bytes, and (as -of October 2013) the inode structure is 156 bytes -(``i_extra_isize = 28``). The extra space between the end of the inode +of August 2019) the inode structure is 160 bytes +(``i_extra_isize = 32``). The extra space between the end of the inode structure and the end of the inode record can be used to store extended attributes. Each inode record can be as large as the filesystem block size, though this is not terribly efficient. diff --git a/Documentation/filesystems/ext4/overview.rst b/Documentation/filesystems/ext4/overview.rst index cbab18baba12..123ebfde47ee 100644 --- a/Documentation/filesystems/ext4/overview.rst +++ b/Documentation/filesystems/ext4/overview.rst @@ -24,3 +24,4 @@ order. .. include:: bigalloc.rst .. include:: inlinedata.rst .. include:: eainode.rst +.. include:: verity.rst diff --git a/Documentation/filesystems/ext4/super.rst b/Documentation/filesystems/ext4/super.rst index 04ff079a2acf..93e55d7c1d40 100644 --- a/Documentation/filesystems/ext4/super.rst +++ b/Documentation/filesystems/ext4/super.rst @@ -58,7 +58,7 @@ The ext4 superblock is laid out as follows in * - 0x1C - \_\_le32 - s\_log\_cluster\_size - - Cluster size is (2 ^ s\_log\_cluster\_size) blocks if bigalloc is + - Cluster size is 2 ^ (10 + s\_log\_cluster\_size) blocks if bigalloc is enabled. Otherwise s\_log\_cluster\_size must equal s\_log\_block\_size. * - 0x20 - \_\_le32 @@ -447,7 +447,7 @@ The ext4 superblock is laid out as follows in - Upper 8 bits of the s_wtime field. * - 0x275 - \_\_u8 - - s\_wtime_hi + - s\_mtime_hi - Upper 8 bits of the s_mtime field. * - 0x276 - \_\_u8 @@ -466,12 +466,20 @@ The ext4 superblock is laid out as follows in - s\_last_error_time_hi - Upper 8 bits of the s_last_error_time_hi field. * - 0x27A - - \_\_u8[2] - - s\_pad + - \_\_u8 + - s\_pad[2] - Zero padding. * - 0x27C + - \_\_le16 + - s\_encoding + - Filename charset encoding. + * - 0x27E + - \_\_le16 + - s\_encoding_flags + - Filename charset encoding flags. + * - 0x280 - \_\_le32 - - s\_reserved[96] + - s\_reserved[95] - Padding to the end of the block. * - 0x3FC - \_\_le32 @@ -617,7 +625,7 @@ following: * - 0x80 - Enable a filesystem size of 2^64 blocks (INCOMPAT\_64BIT). * - 0x100 - - Multiple mount protection. Not implemented (INCOMPAT\_MMP). + - Multiple mount protection (INCOMPAT\_MMP). * - 0x200 - Flexible block groups. See the earlier discussion of this feature (INCOMPAT\_FLEX\_BG). @@ -696,6 +704,8 @@ the following: (RO\_COMPAT\_READONLY) * - 0x2000 - Filesystem tracks project quotas. (RO\_COMPAT\_PROJECT) + * - 0x8000 + - Verity inodes may be present on the filesystem. (RO\_COMPAT\_VERITY) .. _super_def_hash: diff --git a/Documentation/filesystems/ext4/verity.rst b/Documentation/filesystems/ext4/verity.rst new file mode 100644 index 000000000000..3e4c0ee0e068 --- /dev/null +++ b/Documentation/filesystems/ext4/verity.rst @@ -0,0 +1,41 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Verity files +------------ + +ext4 supports fs-verity, which is a filesystem feature that provides +Merkle tree based hashing for individual readonly files. Most of +fs-verity is common to all filesystems that support it; see +:ref:`Documentation/filesystems/fsverity.rst <fsverity>` for the +fs-verity documentation. However, the on-disk layout of the verity +metadata is filesystem-specific. On ext4, the verity metadata is +stored after the end of the file data itself, in the following format: + +- Zero-padding to the next 65536-byte boundary. This padding need not + actually be allocated on-disk, i.e. it may be a hole. + +- The Merkle tree, as documented in + :ref:`Documentation/filesystems/fsverity.rst + <fsverity_merkle_tree>`, with the tree levels stored in order from + root to leaf, and the tree blocks within each level stored in their + natural order. + +- Zero-padding to the next filesystem block boundary. + +- The verity descriptor, as documented in + :ref:`Documentation/filesystems/fsverity.rst <fsverity_descriptor>`, + with optionally appended signature blob. + +- Zero-padding to the next offset that is 4 bytes before a filesystem + block boundary. + +- The size of the verity descriptor in bytes, as a 4-byte little + endian integer. + +Verity inodes have EXT4_VERITY_FL set, and they must use extents, i.e. +EXT4_EXTENTS_FL must be set and EXT4_INLINE_DATA_FL must be clear. +They can have EXT4_ENCRYPT_FL set, in which case the verity metadata +is encrypted as well as the data itself. + +Verity files cannot have blocks allocated past the end of the verity +metadata. diff --git a/Documentation/filesystems/f2fs.txt b/Documentation/filesystems/f2fs.txt index f7b5e4ff0de3..7e1991328473 100644 --- a/Documentation/filesystems/f2fs.txt +++ b/Documentation/filesystems/f2fs.txt @@ -157,6 +157,11 @@ noinline_data Disable the inline data feature, inline data feature is enabled by default. data_flush Enable data flushing before checkpoint in order to persist data of regular and symlink. +reserve_root=%d Support configuring reserved space which is used for + allocation from a privileged user with specified uid or + gid, unit: 4KB, the default limit is 0.2% of user blocks. +resuid=%d The user ID which may use the reserved blocks. +resgid=%d The group ID which may use the reserved blocks. fault_injection=%d Enable fault injection in all supported types with specified injection rate. fault_type=%d Support configuring fault injection type, should be @@ -214,11 +219,22 @@ fsync_mode=%s Control the policy of fsync. Currently supports "posix", non-atomic files likewise "nobarrier" mount option. test_dummy_encryption Enable dummy encryption, which provides a fake fscrypt context. The fake fscrypt context is used by xfstests. -checkpoint=%s Set to "disable" to turn off checkpointing. Set to "enable" +checkpoint=%s[:%u[%]] Set to "disable" to turn off checkpointing. Set to "enable" to reenable checkpointing. Is enabled by default. While disabled, any unmounting or unexpected shutdowns will cause the filesystem contents to appear as they did when the filesystem was mounted with that option. + While mounting with checkpoint=disabled, the filesystem must + run garbage collection to ensure that all available space can + be used. If this takes too much time, the mount may return + EAGAIN. You may optionally add a value to indicate how much + of the disk you would be willing to temporarily give up to + avoid additional garbage collection. This can be given as a + number of blocks, or as a percent. For instance, mounting + with checkpoint=disable:100% would always succeed, but it may + hide up to all remaining free space. The actual space that + would be unusable can be viewed at /sys/fs/f2fs/<disk>/unusable + This space is reclaimed once checkpoint=enable. ================================================================================ DEBUGFS ENTRIES @@ -246,11 +262,14 @@ Files in /sys/fs/f2fs/<devname> .............................................................................. File Content - gc_max_sleep_time This tuning parameter controls the maximum sleep + gc_urgent_sleep_time This parameter controls sleep time for gc_urgent. + 500 ms is set by default. See above gc_urgent. + + gc_min_sleep_time This tuning parameter controls the minimum sleep time for the garbage collection thread. Time is in milliseconds. - gc_min_sleep_time This tuning parameter controls the minimum sleep + gc_max_sleep_time This tuning parameter controls the maximum sleep time for the garbage collection thread. Time is in milliseconds. @@ -270,9 +289,6 @@ Files in /sys/fs/f2fs/<devname> to 1, background thread starts to do GC by given gc_urgent_sleep_time interval. - gc_urgent_sleep_time This parameter controls sleep time for gc_urgent. - 500 ms is set by default. See above gc_urgent. - reclaim_segments This parameter controls the number of prefree segments to be reclaimed. If the number of prefree segments is larger than the number of segments @@ -287,7 +303,16 @@ Files in /sys/fs/f2fs/<devname> checkpoint is triggered, and issued during the checkpoint. By default, it is disabled with 0. - trim_sections This parameter controls the number of sections + discard_granularity This parameter controls the granularity of discard + command size. It will issue discard commands iif + the size is larger than given granularity. Its + unit size is 4KB, and 4 (=16KB) is set by default. + The maximum value is 128 (=512KB). + + reserved_blocks This parameter indicates the number of blocks that + f2fs reserves internally for root. + + batched_trim_sections This parameter controls the number of sections to be trimmed out in batch mode when FITRIM conducts. 32 sections is set by default. @@ -309,11 +334,35 @@ Files in /sys/fs/f2fs/<devname> the number is less than this value, it triggers in-place-updates. + min_seq_blocks This parameter controls the threshold to serialize + write IOs issued by multiple threads in parallel. + + min_hot_blocks This parameter controls the threshold to allocate + a hot data log for pending data blocks to write. + + min_ssr_sections This parameter adds the threshold when deciding + SSR block allocation. If this is large, SSR mode + will be enabled early. + + ram_thresh This parameter controls the memory footprint used + by free nids and cached nat entries. By default, + 10 is set, which indicates 10 MB / 1 GB RAM. + + ra_nid_pages When building free nids, F2FS reads NAT blocks + ahead for speed up. Default is 0. + + dirty_nats_ratio Given dirty ratio of cached nat entries, F2FS + determines flushing them in background. + max_victim_search This parameter controls the number of trials to find a victim segment when conducting SSR and cleaning operations. The default value is 4096 which covers 8GB block address range. + migration_granularity For large-sized sections, F2FS can stop GC given + this granularity instead of reclaiming entire + section. + dir_level This parameter controls the directory level to support large directory. If a directory has a number of files, it can reduce the file lookup @@ -321,9 +370,56 @@ Files in /sys/fs/f2fs/<devname> Otherwise, it needs to decrease this value to reduce the space overhead. The default value is 0. - ram_thresh This parameter controls the memory footprint used - by free nids and cached nat entries. By default, - 10 is set, which indicates 10 MB / 1 GB RAM. + cp_interval F2FS tries to do checkpoint periodically, 60 secs + by default. + + idle_interval F2FS detects system is idle, if there's no F2FS + operations during given interval, 5 secs by + default. + + discard_idle_interval F2FS detects the discard thread is idle, given + time interval. Default is 5 secs. + + gc_idle_interval F2FS detects the GC thread is idle, given time + interval. Default is 5 secs. + + umount_discard_timeout When unmounting the disk, F2FS waits for finishing + queued discard commands which can take huge time. + This gives time out for it, 5 secs by default. + + iostat_enable This controls to enable/disable iostat in F2FS. + + readdir_ra This enables/disabled readahead of inode blocks + in readdir, and default is enabled. + + gc_pin_file_thresh This indicates how many GC can be failed for the + pinned file. If it exceeds this, F2FS doesn't + guarantee its pinning state. 2048 trials is set + by default. + + extension_list This enables to change extension_list for hot/cold + files in runtime. + + inject_rate This controls injection rate of arbitrary faults. + + inject_type This controls injection type of arbitrary faults. + + dirty_segments This shows # of dirty segments. + + lifetime_write_kbytes This shows # of data written to the disk. + + features This shows current features enabled on F2FS. + + current_reserved_blocks This shows # of blocks currently reserved. + + unusable If checkpoint=disable, this shows the number of + blocks that are unusable. + If checkpoint=enable it shows the number of blocks + that would be unusable if checkpoint=disable were + to be set. + +encoding This shows the encoding used for casefolding. + If casefolding is not enabled, returns (none) ================================================================================ USAGE @@ -716,3 +812,28 @@ WRITE_LIFE_NOT_SET WARM_DATA WRITE_LIFE_NOT_SET WRITE_LIFE_NONE " WRITE_LIFE_NONE WRITE_LIFE_MEDIUM " WRITE_LIFE_MEDIUM WRITE_LIFE_LONG " WRITE_LIFE_LONG + +Fallocate(2) Policy +------------------- + +The default policy follows the below posix rule. + +Allocating disk space + The default operation (i.e., mode is zero) of fallocate() allocates + the disk space within the range specified by offset and len. The + file size (as reported by stat(2)) will be changed if offset+len is + greater than the file size. Any subregion within the range specified + by offset and len that did not contain data before the call will be + initialized to zero. This default behavior closely resembles the + behavior of the posix_fallocate(3) library function, and is intended + as a method of optimally implementing that function. + +However, once F2FS receives ioctl(fd, F2FS_IOC_SET_PIN_FILE) in prior to +fallocate(fd, DEFAULT_MODE), it allocates on-disk blocks addressess having +zero or random data, which is useful to the below scenario where: + 1. create(fd) + 2. ioctl(fd, F2FS_IOC_SET_PIN_FILE) + 3. fallocate(fd, 0, 0, size) + 4. address = fibmap(fd, offset) + 5. open(blkdev) + 6. write(blkdev, address) diff --git a/Documentation/filesystems/fscrypt.rst b/Documentation/filesystems/fscrypt.rst index 08c23b60e016..8a0700af9596 100644 --- a/Documentation/filesystems/fscrypt.rst +++ b/Documentation/filesystems/fscrypt.rst @@ -72,6 +72,9 @@ Online attacks fscrypt (and storage encryption in general) can only provide limited protection, if any at all, against online attacks. In detail: +Side-channel attacks +~~~~~~~~~~~~~~~~~~~~ + fscrypt is only resistant to side-channel attacks, such as timing or electromagnetic attacks, to the extent that the underlying Linux Cryptographic API algorithms are. If a vulnerable algorithm is used, @@ -80,29 +83,90 @@ attacker to mount a side channel attack against the online system. Side channel attacks may also be mounted against applications consuming decrypted data. -After an encryption key has been provided, fscrypt is not designed to -hide the plaintext file contents or filenames from other users on the -same system, regardless of the visibility of the keyring key. -Instead, existing access control mechanisms such as file mode bits, -POSIX ACLs, LSMs, or mount namespaces should be used for this purpose. -Also note that as long as the encryption keys are *anywhere* in -memory, an online attacker can necessarily compromise them by mounting -a physical attack or by exploiting any kernel security vulnerability -which provides an arbitrary memory read primitive. - -While it is ostensibly possible to "evict" keys from the system, -recently accessed encrypted files will remain accessible at least -until the filesystem is unmounted or the VFS caches are dropped, e.g. -using ``echo 2 > /proc/sys/vm/drop_caches``. Even after that, if the -RAM is compromised before being powered off, it will likely still be -possible to recover portions of the plaintext file contents, if not -some of the encryption keys as well. (Since Linux v4.12, all -in-kernel keys related to fscrypt are sanitized before being freed. -However, userspace would need to do its part as well.) - -Currently, fscrypt does not prevent a user from maliciously providing -an incorrect key for another user's existing encrypted files. A -protection against this is planned. +Unauthorized file access +~~~~~~~~~~~~~~~~~~~~~~~~ + +After an encryption key has been added, fscrypt does not hide the +plaintext file contents or filenames from other users on the same +system. Instead, existing access control mechanisms such as file mode +bits, POSIX ACLs, LSMs, or namespaces should be used for this purpose. + +(For the reasoning behind this, understand that while the key is +added, the confidentiality of the data, from the perspective of the +system itself, is *not* protected by the mathematical properties of +encryption but rather only by the correctness of the kernel. +Therefore, any encryption-specific access control checks would merely +be enforced by kernel *code* and therefore would be largely redundant +with the wide variety of access control mechanisms already available.) + +Kernel memory compromise +~~~~~~~~~~~~~~~~~~~~~~~~ + +An attacker who compromises the system enough to read from arbitrary +memory, e.g. by mounting a physical attack or by exploiting a kernel +security vulnerability, can compromise all encryption keys that are +currently in use. + +However, fscrypt allows encryption keys to be removed from the kernel, +which may protect them from later compromise. + +In more detail, the FS_IOC_REMOVE_ENCRYPTION_KEY ioctl (or the +FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS ioctl) can wipe a master +encryption key from kernel memory. If it does so, it will also try to +evict all cached inodes which had been "unlocked" using the key, +thereby wiping their per-file keys and making them once again appear +"locked", i.e. in ciphertext or encrypted form. + +However, these ioctls have some limitations: + +- Per-file keys for in-use files will *not* be removed or wiped. + Therefore, for maximum effect, userspace should close the relevant + encrypted files and directories before removing a master key, as + well as kill any processes whose working directory is in an affected + encrypted directory. + +- The kernel cannot magically wipe copies of the master key(s) that + userspace might have as well. Therefore, userspace must wipe all + copies of the master key(s) it makes as well; normally this should + be done immediately after FS_IOC_ADD_ENCRYPTION_KEY, without waiting + for FS_IOC_REMOVE_ENCRYPTION_KEY. Naturally, the same also applies + to all higher levels in the key hierarchy. Userspace should also + follow other security precautions such as mlock()ing memory + containing keys to prevent it from being swapped out. + +- In general, decrypted contents and filenames in the kernel VFS + caches are freed but not wiped. Therefore, portions thereof may be + recoverable from freed memory, even after the corresponding key(s) + were wiped. To partially solve this, you can set + CONFIG_PAGE_POISONING=y in your kernel config and add page_poison=1 + to your kernel command line. However, this has a performance cost. + +- Secret keys might still exist in CPU registers, in crypto + accelerator hardware (if used by the crypto API to implement any of + the algorithms), or in other places not explicitly considered here. + +Limitations of v1 policies +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +v1 encryption policies have some weaknesses with respect to online +attacks: + +- There is no verification that the provided master key is correct. + Therefore, a malicious user can temporarily associate the wrong key + with another user's encrypted files to which they have read-only + access. Because of filesystem caching, the wrong key will then be + used by the other user's accesses to those files, even if the other + user has the correct key in their own keyring. This violates the + meaning of "read-only access". + +- A compromise of a per-file key also compromises the master key from + which it was derived. + +- Non-root users cannot securely remove encryption keys. + +All the above problems are fixed with v2 encryption policies. For +this reason among others, it is recommended to use v2 encryption +policies on all new encrypted directories. Key hierarchy ============= @@ -123,11 +187,52 @@ appropriate master key. There can be any number of master keys, each of which protects any number of directory trees on any number of filesystems. -Userspace should generate master keys either using a cryptographically -secure random number generator, or by using a KDF (Key Derivation -Function). Note that whenever a KDF is used to "stretch" a -lower-entropy secret such as a passphrase, it is critical that a KDF -designed for this purpose be used, such as scrypt, PBKDF2, or Argon2. +Master keys must be real cryptographic keys, i.e. indistinguishable +from random bytestrings of the same length. This implies that users +**must not** directly use a password as a master key, zero-pad a +shorter key, or repeat a shorter key. Security cannot be guaranteed +if userspace makes any such error, as the cryptographic proofs and +analysis would no longer apply. + +Instead, users should generate master keys either using a +cryptographically secure random number generator, or by using a KDF +(Key Derivation Function). The kernel does not do any key stretching; +therefore, if userspace derives the key from a low-entropy secret such +as a passphrase, it is critical that a KDF designed for this purpose +be used, such as scrypt, PBKDF2, or Argon2. + +Key derivation function +----------------------- + +With one exception, fscrypt never uses the master key(s) for +encryption directly. Instead, they are only used as input to a KDF +(Key Derivation Function) to derive the actual keys. + +The KDF used for a particular master key differs depending on whether +the key is used for v1 encryption policies or for v2 encryption +policies. Users **must not** use the same key for both v1 and v2 +encryption policies. (No real-world attack is currently known on this +specific case of key reuse, but its security cannot be guaranteed +since the cryptographic proofs and analysis would no longer apply.) + +For v1 encryption policies, the KDF only supports deriving per-file +encryption keys. It works by encrypting the master key with +AES-128-ECB, using the file's 16-byte nonce as the AES key. The +resulting ciphertext is used as the derived key. If the ciphertext is +longer than needed, then it is truncated to the needed length. + +For v2 encryption policies, the KDF is HKDF-SHA512. The master key is +passed as the "input keying material", no salt is used, and a distinct +"application-specific information string" is used for each distinct +key to be derived. For example, when a per-file encryption key is +derived, the application-specific information string is the file's +nonce prefixed with "fscrypt\\0" and a context byte. Different +context bytes are used for other types of derived keys. + +HKDF-SHA512 is preferred to the original AES-128-ECB based KDF because +HKDF is more flexible, is nonreversible, and evenly distributes +entropy from the master key. HKDF is also standardized and widely +used by other software, whereas the AES-128-ECB based KDF is ad-hoc. Per-file keys ------------- @@ -138,29 +243,9 @@ files doesn't map to the same ciphertext, or vice versa. In most cases, fscrypt does this by deriving per-file keys. When a new encrypted inode (regular file, directory, or symlink) is created, fscrypt randomly generates a 16-byte nonce and stores it in the -inode's encryption xattr. Then, it uses a KDF (Key Derivation -Function) to derive the file's key from the master key and nonce. - -The Adiantum encryption mode (see `Encryption modes and usage`_) is -special, since it accepts longer IVs and is suitable for both contents -and filenames encryption. For it, a "direct key" option is offered -where the file's nonce is included in the IVs and the master key is -used for encryption directly. This improves performance; however, -users must not use the same master key for any other encryption mode. - -Below, the KDF and design considerations are described in more detail. - -The current KDF works by encrypting the master key with AES-128-ECB, -using the file's nonce as the AES key. The output is used as the -derived key. If the output is longer than needed, then it is -truncated to the needed length. - -Note: this KDF meets the primary security requirement, which is to -produce unique derived keys that preserve the entropy of the master -key, assuming that the master key is already a good pseudorandom key. -However, it is nonstandard and has some problems such as being -reversible, so it is generally considered to be a mistake! It may be -replaced with HKDF or another more standard KDF in the future. +inode's encryption xattr. Then, it uses a KDF (as described in `Key +derivation function`_) to derive the file's key from the master key +and nonce. Key derivation was chosen over key wrapping because wrapped keys would require larger xattrs which would be less likely to fit in-line in the @@ -176,6 +261,37 @@ rejected as it would have prevented ext4 filesystems from being resized, and by itself still wouldn't have been sufficient to prevent the same key from being directly reused for both XTS and CTS-CBC. +DIRECT_KEY and per-mode keys +---------------------------- + +The Adiantum encryption mode (see `Encryption modes and usage`_) is +suitable for both contents and filenames encryption, and it accepts +long IVs --- long enough to hold both an 8-byte logical block number +and a 16-byte per-file nonce. Also, the overhead of each Adiantum key +is greater than that of an AES-256-XTS key. + +Therefore, to improve performance and save memory, for Adiantum a +"direct key" configuration is supported. When the user has enabled +this by setting FSCRYPT_POLICY_FLAG_DIRECT_KEY in the fscrypt policy, +per-file keys are not used. Instead, whenever any data (contents or +filenames) is encrypted, the file's 16-byte nonce is included in the +IV. Moreover: + +- For v1 encryption policies, the encryption is done directly with the + master key. Because of this, users **must not** use the same master + key for any other purpose, even for other v1 policies. + +- For v2 encryption policies, the encryption is done with a per-mode + key derived using the KDF. Users may use the same master key for + other v2 encryption policies. + +Key identifiers +--------------- + +For master keys used for v2 encryption policies, a unique 16-byte "key +identifier" is also derived using the KDF. This value is stored in +the clear, since it is needed to reliably identify the key itself. + Encryption modes and usage ========================== @@ -191,7 +307,9 @@ Currently, the following pairs of encryption modes are supported: If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair. AES-128-CBC was added only for low-powered embedded devices with -crypto accelerators such as CAAM or CESA that do not support XTS. +crypto accelerators such as CAAM or CESA that do not support XTS. To +use AES-128-CBC, CONFIG_CRYPTO_SHA256 (or another SHA-256 +implementation) must be enabled so that ESSIV can be used. Adiantum is a (primarily) stream cipher-based mode that is fast even on CPUs without dedicated crypto instructions. It's also a true @@ -223,9 +341,10 @@ a little endian number, except that: is encrypted with AES-256 where the AES-256 key is the SHA-256 hash of the file's data encryption key. -- In the "direct key" configuration (FS_POLICY_FLAG_DIRECT_KEY set in - the fscrypt_policy), the file's nonce is also appended to the IV. - Currently this is only allowed with the Adiantum encryption mode. +- In the "direct key" configuration (FSCRYPT_POLICY_FLAG_DIRECT_KEY + set in the fscrypt_policy), the file's nonce is also appended to the + IV. Currently this is only allowed with the Adiantum encryption + mode. Filenames encryption -------------------- @@ -267,49 +386,77 @@ User API Setting an encryption policy ---------------------------- +FS_IOC_SET_ENCRYPTION_POLICY +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + The FS_IOC_SET_ENCRYPTION_POLICY ioctl sets an encryption policy on an empty directory or verifies that a directory or regular file already has the specified encryption policy. It takes in a pointer to a -:c:type:`struct fscrypt_policy`, defined as follows:: +:c:type:`struct fscrypt_policy_v1` or a :c:type:`struct +fscrypt_policy_v2`, defined as follows:: - #define FS_KEY_DESCRIPTOR_SIZE 8 + #define FSCRYPT_POLICY_V1 0 + #define FSCRYPT_KEY_DESCRIPTOR_SIZE 8 + struct fscrypt_policy_v1 { + __u8 version; + __u8 contents_encryption_mode; + __u8 filenames_encryption_mode; + __u8 flags; + __u8 master_key_descriptor[FSCRYPT_KEY_DESCRIPTOR_SIZE]; + }; + #define fscrypt_policy fscrypt_policy_v1 - struct fscrypt_policy { + #define FSCRYPT_POLICY_V2 2 + #define FSCRYPT_KEY_IDENTIFIER_SIZE 16 + struct fscrypt_policy_v2 { __u8 version; __u8 contents_encryption_mode; __u8 filenames_encryption_mode; __u8 flags; - __u8 master_key_descriptor[FS_KEY_DESCRIPTOR_SIZE]; + __u8 __reserved[4]; + __u8 master_key_identifier[FSCRYPT_KEY_IDENTIFIER_SIZE]; }; This structure must be initialized as follows: -- ``version`` must be 0. +- ``version`` must be FSCRYPT_POLICY_V1 (0) if the struct is + :c:type:`fscrypt_policy_v1` or FSCRYPT_POLICY_V2 (2) if the struct + is :c:type:`fscrypt_policy_v2`. (Note: we refer to the original + policy version as "v1", though its version code is really 0.) For + new encrypted directories, use v2 policies. - ``contents_encryption_mode`` and ``filenames_encryption_mode`` must - be set to constants from ``<linux/fs.h>`` which identify the - encryption modes to use. If unsure, use - FS_ENCRYPTION_MODE_AES_256_XTS (1) for ``contents_encryption_mode`` - and FS_ENCRYPTION_MODE_AES_256_CTS (4) for - ``filenames_encryption_mode``. + be set to constants from ``<linux/fscrypt.h>`` which identify the + encryption modes to use. If unsure, use FSCRYPT_MODE_AES_256_XTS + (1) for ``contents_encryption_mode`` and FSCRYPT_MODE_AES_256_CTS + (4) for ``filenames_encryption_mode``. -- ``flags`` must contain a value from ``<linux/fs.h>`` which +- ``flags`` must contain a value from ``<linux/fscrypt.h>`` which identifies the amount of NUL-padding to use when encrypting - filenames. If unsure, use FS_POLICY_FLAGS_PAD_32 (0x3). - In addition, if the chosen encryption modes are both - FS_ENCRYPTION_MODE_ADIANTUM, this can contain - FS_POLICY_FLAG_DIRECT_KEY to specify that the master key should be - used directly, without key derivation. - -- ``master_key_descriptor`` specifies how to find the master key in - the keyring; see `Adding keys`_. It is up to userspace to choose a - unique ``master_key_descriptor`` for each master key. The e4crypt - and fscrypt tools use the first 8 bytes of + filenames. If unsure, use FSCRYPT_POLICY_FLAGS_PAD_32 (0x3). + Additionally, if the encryption modes are both + FSCRYPT_MODE_ADIANTUM, this can contain + FSCRYPT_POLICY_FLAG_DIRECT_KEY; see `DIRECT_KEY and per-mode keys`_. + +- For v2 encryption policies, ``__reserved`` must be zeroed. + +- For v1 encryption policies, ``master_key_descriptor`` specifies how + to find the master key in a keyring; see `Adding keys`_. It is up + to userspace to choose a unique ``master_key_descriptor`` for each + master key. The e4crypt and fscrypt tools use the first 8 bytes of ``SHA-512(SHA-512(master_key))``, but this particular scheme is not required. Also, the master key need not be in the keyring yet when FS_IOC_SET_ENCRYPTION_POLICY is executed. However, it must be added before any files can be created in the encrypted directory. + For v2 encryption policies, ``master_key_descriptor`` has been + replaced with ``master_key_identifier``, which is longer and cannot + be arbitrarily chosen. Instead, the key must first be added using + `FS_IOC_ADD_ENCRYPTION_KEY`_. Then, the ``key_spec.u.identifier`` + the kernel returned in the :c:type:`struct fscrypt_add_key_arg` must + be used as the ``master_key_identifier`` in the :c:type:`struct + fscrypt_policy_v2`. + If the file is not yet encrypted, then FS_IOC_SET_ENCRYPTION_POLICY verifies that the file is an empty directory. If so, the specified encryption policy is assigned to the directory, turning it into an @@ -325,6 +472,15 @@ policy exactly matches the actual one. If they match, then the ioctl returns 0. Otherwise, it fails with EEXIST. This works on both regular files and directories, including nonempty directories. +When a v2 encryption policy is assigned to a directory, it is also +required that either the specified key has been added by the current +user or that the caller has CAP_FOWNER in the initial user namespace. +(This is needed to prevent a user from encrypting their data with +another user's key.) The key must remain added while +FS_IOC_SET_ENCRYPTION_POLICY is executing. However, if the new +encrypted directory does not need to be accessed immediately, then the +key can be removed right away afterwards. + Note that the ext4 filesystem does not allow the root directory to be encrypted, even if it is empty. Users who want to encrypt an entire filesystem with one key should consider using dm-crypt instead. @@ -337,7 +493,11 @@ FS_IOC_SET_ENCRYPTION_POLICY can fail with the following errors: - ``EEXIST``: the file is already encrypted with an encryption policy different from the one specified - ``EINVAL``: an invalid encryption policy was specified (invalid - version, mode(s), or flags) + version, mode(s), or flags; or reserved bits were set) +- ``ENOKEY``: a v2 encryption policy was specified, but the key with + the specified ``master_key_identifier`` has not been added, nor does + the process have the CAP_FOWNER capability in the initial user + namespace - ``ENOTDIR``: the file is unencrypted and is a regular file, not a directory - ``ENOTEMPTY``: the file is unencrypted and is a nonempty directory @@ -356,25 +516,79 @@ FS_IOC_SET_ENCRYPTION_POLICY can fail with the following errors: Getting an encryption policy ---------------------------- -The FS_IOC_GET_ENCRYPTION_POLICY ioctl retrieves the :c:type:`struct -fscrypt_policy`, if any, for a directory or regular file. See above -for the struct definition. No additional permissions are required -beyond the ability to open the file. +Two ioctls are available to get a file's encryption policy: + +- `FS_IOC_GET_ENCRYPTION_POLICY_EX`_ +- `FS_IOC_GET_ENCRYPTION_POLICY`_ + +The extended (_EX) version of the ioctl is more general and is +recommended to use when possible. However, on older kernels only the +original ioctl is available. Applications should try the extended +version, and if it fails with ENOTTY fall back to the original +version. + +FS_IOC_GET_ENCRYPTION_POLICY_EX +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The FS_IOC_GET_ENCRYPTION_POLICY_EX ioctl retrieves the encryption +policy, if any, for a directory or regular file. No additional +permissions are required beyond the ability to open the file. It +takes in a pointer to a :c:type:`struct fscrypt_get_policy_ex_arg`, +defined as follows:: + + struct fscrypt_get_policy_ex_arg { + __u64 policy_size; /* input/output */ + union { + __u8 version; + struct fscrypt_policy_v1 v1; + struct fscrypt_policy_v2 v2; + } policy; /* output */ + }; + +The caller must initialize ``policy_size`` to the size available for +the policy struct, i.e. ``sizeof(arg.policy)``. -FS_IOC_GET_ENCRYPTION_POLICY can fail with the following errors: +On success, the policy struct is returned in ``policy``, and its +actual size is returned in ``policy_size``. ``policy.version`` should +be checked to determine the version of policy returned. Note that the +version code for the "v1" policy is actually 0 (FSCRYPT_POLICY_V1). + +FS_IOC_GET_ENCRYPTION_POLICY_EX can fail with the following errors: - ``EINVAL``: the file is encrypted, but it uses an unrecognized - encryption context format + encryption policy version - ``ENODATA``: the file is not encrypted -- ``ENOTTY``: this type of filesystem does not implement encryption +- ``ENOTTY``: this type of filesystem does not implement encryption, + or this kernel is too old to support FS_IOC_GET_ENCRYPTION_POLICY_EX + (try FS_IOC_GET_ENCRYPTION_POLICY instead) - ``EOPNOTSUPP``: the kernel was not configured with encryption - support for this filesystem + support for this filesystem, or the filesystem superblock has not + had encryption enabled on it +- ``EOVERFLOW``: the file is encrypted and uses a recognized + encryption policy version, but the policy struct does not fit into + the provided buffer Note: if you only need to know whether a file is encrypted or not, on most filesystems it is also possible to use the FS_IOC_GETFLAGS ioctl and check for FS_ENCRYPT_FL, or to use the statx() system call and check for STATX_ATTR_ENCRYPTED in stx_attributes. +FS_IOC_GET_ENCRYPTION_POLICY +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The FS_IOC_GET_ENCRYPTION_POLICY ioctl can also retrieve the +encryption policy, if any, for a directory or regular file. However, +unlike `FS_IOC_GET_ENCRYPTION_POLICY_EX`_, +FS_IOC_GET_ENCRYPTION_POLICY only supports the original policy +version. It takes in a pointer directly to a :c:type:`struct +fscrypt_policy_v1` rather than a :c:type:`struct +fscrypt_get_policy_ex_arg`. + +The error codes for FS_IOC_GET_ENCRYPTION_POLICY are the same as those +for FS_IOC_GET_ENCRYPTION_POLICY_EX, except that +FS_IOC_GET_ENCRYPTION_POLICY also returns ``EINVAL`` if the file is +encrypted using a newer encryption policy version. + Getting the per-filesystem salt ------------------------------- @@ -390,8 +604,115 @@ generate and manage any needed salt(s) in userspace. Adding keys ----------- -To provide a master key, userspace must add it to an appropriate -keyring using the add_key() system call (see: +FS_IOC_ADD_ENCRYPTION_KEY +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The FS_IOC_ADD_ENCRYPTION_KEY ioctl adds a master encryption key to +the filesystem, making all files on the filesystem which were +encrypted using that key appear "unlocked", i.e. in plaintext form. +It can be executed on any file or directory on the target filesystem, +but using the filesystem's root directory is recommended. It takes in +a pointer to a :c:type:`struct fscrypt_add_key_arg`, defined as +follows:: + + struct fscrypt_add_key_arg { + struct fscrypt_key_specifier key_spec; + __u32 raw_size; + __u32 __reserved[9]; + __u8 raw[]; + }; + + #define FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR 1 + #define FSCRYPT_KEY_SPEC_TYPE_IDENTIFIER 2 + + struct fscrypt_key_specifier { + __u32 type; /* one of FSCRYPT_KEY_SPEC_TYPE_* */ + __u32 __reserved; + union { + __u8 __reserved[32]; /* reserve some extra space */ + __u8 descriptor[FSCRYPT_KEY_DESCRIPTOR_SIZE]; + __u8 identifier[FSCRYPT_KEY_IDENTIFIER_SIZE]; + } u; + }; + +:c:type:`struct fscrypt_add_key_arg` must be zeroed, then initialized +as follows: + +- If the key is being added for use by v1 encryption policies, then + ``key_spec.type`` must contain FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR, and + ``key_spec.u.descriptor`` must contain the descriptor of the key + being added, corresponding to the value in the + ``master_key_descriptor`` field of :c:type:`struct + fscrypt_policy_v1`. To add this type of key, the calling process + must have the CAP_SYS_ADMIN capability in the initial user + namespace. + + Alternatively, if the key is being added for use by v2 encryption + policies, then ``key_spec.type`` must contain + FSCRYPT_KEY_SPEC_TYPE_IDENTIFIER, and ``key_spec.u.identifier`` is + an *output* field which the kernel fills in with a cryptographic + hash of the key. To add this type of key, the calling process does + not need any privileges. However, the number of keys that can be + added is limited by the user's quota for the keyrings service (see + ``Documentation/security/keys/core.rst``). + +- ``raw_size`` must be the size of the ``raw`` key provided, in bytes. + +- ``raw`` is a variable-length field which must contain the actual + key, ``raw_size`` bytes long. + +For v2 policy keys, the kernel keeps track of which user (identified +by effective user ID) added the key, and only allows the key to be +removed by that user --- or by "root", if they use +`FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS`_. + +However, if another user has added the key, it may be desirable to +prevent that other user from unexpectedly removing it. Therefore, +FS_IOC_ADD_ENCRYPTION_KEY may also be used to add a v2 policy key +*again*, even if it's already added by other user(s). In this case, +FS_IOC_ADD_ENCRYPTION_KEY will just install a claim to the key for the +current user, rather than actually add the key again (but the raw key +must still be provided, as a proof of knowledge). + +FS_IOC_ADD_ENCRYPTION_KEY returns 0 if either the key or a claim to +the key was either added or already exists. + +FS_IOC_ADD_ENCRYPTION_KEY can fail with the following errors: + +- ``EACCES``: FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR was specified, but the + caller does not have the CAP_SYS_ADMIN capability in the initial + user namespace +- ``EDQUOT``: the key quota for this user would be exceeded by adding + the key +- ``EINVAL``: invalid key size or key specifier type, or reserved bits + were set +- ``ENOTTY``: this type of filesystem does not implement encryption +- ``EOPNOTSUPP``: the kernel was not configured with encryption + support for this filesystem, or the filesystem superblock has not + had encryption enabled on it + +Legacy method +~~~~~~~~~~~~~ + +For v1 encryption policies, a master encryption key can also be +provided by adding it to a process-subscribed keyring, e.g. to a +session keyring, or to a user keyring if the user keyring is linked +into the session keyring. + +This method is deprecated (and not supported for v2 encryption +policies) for several reasons. First, it cannot be used in +combination with FS_IOC_REMOVE_ENCRYPTION_KEY (see `Removing keys`_), +so for removing a key a workaround such as keyctl_unlink() in +combination with ``sync; echo 2 > /proc/sys/vm/drop_caches`` would +have to be used. Second, it doesn't match the fact that the +locked/unlocked status of encrypted files (i.e. whether they appear to +be in plaintext form or in ciphertext form) is global. This mismatch +has caused much confusion as well as real problems when processes +running under different UIDs, such as a ``sudo`` command, need to +access encrypted files. + +Nevertheless, to add a key to one of the process-subscribed keyrings, +the add_key() system call can be used (see: ``Documentation/security/keys/core.rst``). The key type must be "logon"; keys of this type are kept in kernel memory and cannot be read back by userspace. The key description must be "fscrypt:" @@ -399,12 +720,12 @@ followed by the 16-character lower case hex representation of the ``master_key_descriptor`` that was set in the encryption policy. The key payload must conform to the following structure:: - #define FS_MAX_KEY_SIZE 64 + #define FSCRYPT_MAX_KEY_SIZE 64 struct fscrypt_key { - u32 mode; - u8 raw[FS_MAX_KEY_SIZE]; - u32 size; + __u32 mode; + __u8 raw[FSCRYPT_MAX_KEY_SIZE]; + __u32 size; }; ``mode`` is ignored; just set it to 0. The actual key is provided in @@ -416,26 +737,194 @@ with a filesystem-specific prefix such as "ext4:". However, the filesystem-specific prefixes are deprecated and should not be used in new programs. -There are several different types of keyrings in which encryption keys -may be placed, such as a session keyring, a user session keyring, or a -user keyring. Each key must be placed in a keyring that is "attached" -to all processes that might need to access files encrypted with it, in -the sense that request_key() will find the key. Generally, if only -processes belonging to a specific user need to access a given -encrypted directory and no session keyring has been installed, then -that directory's key should be placed in that user's user session -keyring or user keyring. Otherwise, a session keyring should be -installed if needed, and the key should be linked into that session -keyring, or in a keyring linked into that session keyring. - -Note: introducing the complex visibility semantics of keyrings here -was arguably a mistake --- especially given that by design, after any -process successfully opens an encrypted file (thereby setting up the -per-file key), possessing the keyring key is not actually required for -any process to read/write the file until its in-memory inode is -evicted. In the future there probably should be a way to provide keys -directly to the filesystem instead, which would make the intended -semantics clearer. +Removing keys +------------- + +Two ioctls are available for removing a key that was added by +`FS_IOC_ADD_ENCRYPTION_KEY`_: + +- `FS_IOC_REMOVE_ENCRYPTION_KEY`_ +- `FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS`_ + +These two ioctls differ only in cases where v2 policy keys are added +or removed by non-root users. + +These ioctls don't work on keys that were added via the legacy +process-subscribed keyrings mechanism. + +Before using these ioctls, read the `Kernel memory compromise`_ +section for a discussion of the security goals and limitations of +these ioctls. + +FS_IOC_REMOVE_ENCRYPTION_KEY +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The FS_IOC_REMOVE_ENCRYPTION_KEY ioctl removes a claim to a master +encryption key from the filesystem, and possibly removes the key +itself. It can be executed on any file or directory on the target +filesystem, but using the filesystem's root directory is recommended. +It takes in a pointer to a :c:type:`struct fscrypt_remove_key_arg`, +defined as follows:: + + struct fscrypt_remove_key_arg { + struct fscrypt_key_specifier key_spec; + #define FSCRYPT_KEY_REMOVAL_STATUS_FLAG_FILES_BUSY 0x00000001 + #define FSCRYPT_KEY_REMOVAL_STATUS_FLAG_OTHER_USERS 0x00000002 + __u32 removal_status_flags; /* output */ + __u32 __reserved[5]; + }; + +This structure must be zeroed, then initialized as follows: + +- The key to remove is specified by ``key_spec``: + + - To remove a key used by v1 encryption policies, set + ``key_spec.type`` to FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR and fill + in ``key_spec.u.descriptor``. To remove this type of key, the + calling process must have the CAP_SYS_ADMIN capability in the + initial user namespace. + + - To remove a key used by v2 encryption policies, set + ``key_spec.type`` to FSCRYPT_KEY_SPEC_TYPE_IDENTIFIER and fill + in ``key_spec.u.identifier``. + +For v2 policy keys, this ioctl is usable by non-root users. However, +to make this possible, it actually just removes the current user's +claim to the key, undoing a single call to FS_IOC_ADD_ENCRYPTION_KEY. +Only after all claims are removed is the key really removed. + +For example, if FS_IOC_ADD_ENCRYPTION_KEY was called with uid 1000, +then the key will be "claimed" by uid 1000, and +FS_IOC_REMOVE_ENCRYPTION_KEY will only succeed as uid 1000. Or, if +both uids 1000 and 2000 added the key, then for each uid +FS_IOC_REMOVE_ENCRYPTION_KEY will only remove their own claim. Only +once *both* are removed is the key really removed. (Think of it like +unlinking a file that may have hard links.) + +If FS_IOC_REMOVE_ENCRYPTION_KEY really removes the key, it will also +try to "lock" all files that had been unlocked with the key. It won't +lock files that are still in-use, so this ioctl is expected to be used +in cooperation with userspace ensuring that none of the files are +still open. However, if necessary, this ioctl can be executed again +later to retry locking any remaining files. + +FS_IOC_REMOVE_ENCRYPTION_KEY returns 0 if either the key was removed +(but may still have files remaining to be locked), the user's claim to +the key was removed, or the key was already removed but had files +remaining to be the locked so the ioctl retried locking them. In any +of these cases, ``removal_status_flags`` is filled in with the +following informational status flags: + +- ``FSCRYPT_KEY_REMOVAL_STATUS_FLAG_FILES_BUSY``: set if some file(s) + are still in-use. Not guaranteed to be set in the case where only + the user's claim to the key was removed. +- ``FSCRYPT_KEY_REMOVAL_STATUS_FLAG_OTHER_USERS``: set if only the + user's claim to the key was removed, not the key itself + +FS_IOC_REMOVE_ENCRYPTION_KEY can fail with the following errors: + +- ``EACCES``: The FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR key specifier type + was specified, but the caller does not have the CAP_SYS_ADMIN + capability in the initial user namespace +- ``EINVAL``: invalid key specifier type, or reserved bits were set +- ``ENOKEY``: the key object was not found at all, i.e. it was never + added in the first place or was already fully removed including all + files locked; or, the user does not have a claim to the key (but + someone else does). +- ``ENOTTY``: this type of filesystem does not implement encryption +- ``EOPNOTSUPP``: the kernel was not configured with encryption + support for this filesystem, or the filesystem superblock has not + had encryption enabled on it + +FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS is exactly the same as +`FS_IOC_REMOVE_ENCRYPTION_KEY`_, except that for v2 policy keys, the +ALL_USERS version of the ioctl will remove all users' claims to the +key, not just the current user's. I.e., the key itself will always be +removed, no matter how many users have added it. This difference is +only meaningful if non-root users are adding and removing keys. + +Because of this, FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS also requires +"root", namely the CAP_SYS_ADMIN capability in the initial user +namespace. Otherwise it will fail with EACCES. + +Getting key status +------------------ + +FS_IOC_GET_ENCRYPTION_KEY_STATUS +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The FS_IOC_GET_ENCRYPTION_KEY_STATUS ioctl retrieves the status of a +master encryption key. It can be executed on any file or directory on +the target filesystem, but using the filesystem's root directory is +recommended. It takes in a pointer to a :c:type:`struct +fscrypt_get_key_status_arg`, defined as follows:: + + struct fscrypt_get_key_status_arg { + /* input */ + struct fscrypt_key_specifier key_spec; + __u32 __reserved[6]; + + /* output */ + #define FSCRYPT_KEY_STATUS_ABSENT 1 + #define FSCRYPT_KEY_STATUS_PRESENT 2 + #define FSCRYPT_KEY_STATUS_INCOMPLETELY_REMOVED 3 + __u32 status; + #define FSCRYPT_KEY_STATUS_FLAG_ADDED_BY_SELF 0x00000001 + __u32 status_flags; + __u32 user_count; + __u32 __out_reserved[13]; + }; + +The caller must zero all input fields, then fill in ``key_spec``: + + - To get the status of a key for v1 encryption policies, set + ``key_spec.type`` to FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR and fill + in ``key_spec.u.descriptor``. + + - To get the status of a key for v2 encryption policies, set + ``key_spec.type`` to FSCRYPT_KEY_SPEC_TYPE_IDENTIFIER and fill + in ``key_spec.u.identifier``. + +On success, 0 is returned and the kernel fills in the output fields: + +- ``status`` indicates whether the key is absent, present, or + incompletely removed. Incompletely removed means that the master + secret has been removed, but some files are still in use; i.e., + `FS_IOC_REMOVE_ENCRYPTION_KEY`_ returned 0 but set the informational + status flag FSCRYPT_KEY_REMOVAL_STATUS_FLAG_FILES_BUSY. + +- ``status_flags`` can contain the following flags: + + - ``FSCRYPT_KEY_STATUS_FLAG_ADDED_BY_SELF`` indicates that the key + has added by the current user. This is only set for keys + identified by ``identifier`` rather than by ``descriptor``. + +- ``user_count`` specifies the number of users who have added the key. + This is only set for keys identified by ``identifier`` rather than + by ``descriptor``. + +FS_IOC_GET_ENCRYPTION_KEY_STATUS can fail with the following errors: + +- ``EINVAL``: invalid key specifier type, or reserved bits were set +- ``ENOTTY``: this type of filesystem does not implement encryption +- ``EOPNOTSUPP``: the kernel was not configured with encryption + support for this filesystem, or the filesystem superblock has not + had encryption enabled on it + +Among other use cases, FS_IOC_GET_ENCRYPTION_KEY_STATUS can be useful +for determining whether the key for a given encrypted directory needs +to be added before prompting the user for the passphrase needed to +derive the key. + +FS_IOC_GET_ENCRYPTION_KEY_STATUS can only get the status of keys in +the filesystem-level keyring, i.e. the keyring managed by +`FS_IOC_ADD_ENCRYPTION_KEY`_ and `FS_IOC_REMOVE_ENCRYPTION_KEY`_. It +cannot get the status of a key that has only been added for use by v1 +encryption policies using the legacy mechanism involving +process-subscribed keyrings. Access semantics ================ @@ -498,7 +987,7 @@ Without the key Some filesystem operations may be performed on encrypted regular files, directories, and symlinks even before their encryption key has -been provided: +been added, or after their encryption key has been removed: - File metadata may be read, e.g. using stat(). @@ -563,33 +1052,44 @@ Encryption context ------------------ An encryption policy is represented on-disk by a :c:type:`struct -fscrypt_context`. It is up to individual filesystems to decide where -to store it, but normally it would be stored in a hidden extended -attribute. It should *not* be exposed by the xattr-related system -calls such as getxattr() and setxattr() because of the special -semantics of the encryption xattr. (In particular, there would be -much confusion if an encryption policy were to be added to or removed -from anything other than an empty directory.) The struct is defined -as follows:: - - #define FS_KEY_DESCRIPTOR_SIZE 8 +fscrypt_context_v1` or a :c:type:`struct fscrypt_context_v2`. It is +up to individual filesystems to decide where to store it, but normally +it would be stored in a hidden extended attribute. It should *not* be +exposed by the xattr-related system calls such as getxattr() and +setxattr() because of the special semantics of the encryption xattr. +(In particular, there would be much confusion if an encryption policy +were to be added to or removed from anything other than an empty +directory.) These structs are defined as follows:: + #define FS_KEY_DERIVATION_NONCE_SIZE 16 - struct fscrypt_context { - u8 format; + #define FSCRYPT_KEY_DESCRIPTOR_SIZE 8 + struct fscrypt_context_v1 { + u8 version; + u8 contents_encryption_mode; + u8 filenames_encryption_mode; + u8 flags; + u8 master_key_descriptor[FSCRYPT_KEY_DESCRIPTOR_SIZE]; + u8 nonce[FS_KEY_DERIVATION_NONCE_SIZE]; + }; + + #define FSCRYPT_KEY_IDENTIFIER_SIZE 16 + struct fscrypt_context_v2 { + u8 version; u8 contents_encryption_mode; u8 filenames_encryption_mode; u8 flags; - u8 master_key_descriptor[FS_KEY_DESCRIPTOR_SIZE]; + u8 __reserved[4]; + u8 master_key_identifier[FSCRYPT_KEY_IDENTIFIER_SIZE]; u8 nonce[FS_KEY_DERIVATION_NONCE_SIZE]; }; -Note that :c:type:`struct fscrypt_context` contains the same -information as :c:type:`struct fscrypt_policy` (see `Setting an -encryption policy`_), except that :c:type:`struct fscrypt_context` -also contains a nonce. The nonce is randomly generated by the kernel -and is used to derive the inode's encryption key as described in -`Per-file keys`_. +The context structs contain the same information as the corresponding +policy structs (see `Setting an encryption policy`_), except that the +context structs also contain a nonce. The nonce is randomly generated +by the kernel and is used as KDF input or as a tweak to cause +different files to be encrypted differently; see `Per-file keys`_ and +`DIRECT_KEY and per-mode keys`_. Data path changes ----------------- @@ -647,3 +1147,42 @@ Note that the precise way that filenames are presented to userspace without the key is subject to change in the future. It is only meant as a way to temporarily present valid filenames so that commands like ``rm -r`` work as expected on encrypted directories. + +Tests +===== + +To test fscrypt, use xfstests, which is Linux's de facto standard +filesystem test suite. First, run all the tests in the "encrypt" +group on the relevant filesystem(s). For example, to test ext4 and +f2fs encryption using `kvm-xfstests +<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_:: + + kvm-xfstests -c ext4,f2fs -g encrypt + +UBIFS encryption can also be tested this way, but it should be done in +a separate command, and it takes some time for kvm-xfstests to set up +emulated UBI volumes:: + + kvm-xfstests -c ubifs -g encrypt + +No tests should fail. However, tests that use non-default encryption +modes (e.g. generic/549 and generic/550) will be skipped if the needed +algorithms were not built into the kernel's crypto API. Also, tests +that access the raw block device (e.g. generic/399, generic/548, +generic/549, generic/550) will be skipped on UBIFS. + +Besides running the "encrypt" group tests, for ext4 and f2fs it's also +possible to run most xfstests with the "test_dummy_encryption" mount +option. This option causes all new files to be automatically +encrypted with a dummy key, without having to make any API calls. +This tests the encrypted I/O paths more thoroughly. To do this with +kvm-xfstests, use the "encrypt" filesystem configuration:: + + kvm-xfstests -c ext4/encrypt,f2fs/encrypt -g auto + +Because this runs many more tests than "-g encrypt" does, it takes +much longer to run; so also consider using `gce-xfstests +<https://github.com/tytso/xfstests-bld/blob/master/Documentation/gce-xfstests.md>`_ +instead of kvm-xfstests:: + + gce-xfstests -c ext4/encrypt,f2fs/encrypt -g auto diff --git a/Documentation/filesystems/fsverity.rst b/Documentation/filesystems/fsverity.rst new file mode 100644 index 000000000000..42a0b6dd9e0b --- /dev/null +++ b/Documentation/filesystems/fsverity.rst @@ -0,0 +1,726 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _fsverity: + +======================================================= +fs-verity: read-only file-based authenticity protection +======================================================= + +Introduction +============ + +fs-verity (``fs/verity/``) is a support layer that filesystems can +hook into to support transparent integrity and authenticity protection +of read-only files. Currently, it is supported by the ext4 and f2fs +filesystems. Like fscrypt, not too much filesystem-specific code is +needed to support fs-verity. + +fs-verity is similar to `dm-verity +<https://www.kernel.org/doc/Documentation/device-mapper/verity.txt>`_ +but works on files rather than block devices. On regular files on +filesystems supporting fs-verity, userspace can execute an ioctl that +causes the filesystem to build a Merkle tree for the file and persist +it to a filesystem-specific location associated with the file. + +After this, the file is made readonly, and all reads from the file are +automatically verified against the file's Merkle tree. Reads of any +corrupted data, including mmap reads, will fail. + +Userspace can use another ioctl to retrieve the root hash (actually +the "file measurement", which is a hash that includes the root hash) +that fs-verity is enforcing for the file. This ioctl executes in +constant time, regardless of the file size. + +fs-verity is essentially a way to hash a file in constant time, +subject to the caveat that reads which would violate the hash will +fail at runtime. + +Use cases +========= + +By itself, the base fs-verity feature only provides integrity +protection, i.e. detection of accidental (non-malicious) corruption. + +However, because fs-verity makes retrieving the file hash extremely +efficient, it's primarily meant to be used as a tool to support +authentication (detection of malicious modifications) or auditing +(logging file hashes before use). + +Trusted userspace code (e.g. operating system code running on a +read-only partition that is itself authenticated by dm-verity) can +authenticate the contents of an fs-verity file by using the +`FS_IOC_MEASURE_VERITY`_ ioctl to retrieve its hash, then verifying a +digital signature of it. + +A standard file hash could be used instead of fs-verity. However, +this is inefficient if the file is large and only a small portion may +be accessed. This is often the case for Android application package +(APK) files, for example. These typically contain many translations, +classes, and other resources that are infrequently or even never +accessed on a particular device. It would be slow and wasteful to +read and hash the entire file before starting the application. + +Unlike an ahead-of-time hash, fs-verity also re-verifies data each +time it's paged in. This ensures that malicious disk firmware can't +undetectably change the contents of the file at runtime. + +fs-verity does not replace or obsolete dm-verity. dm-verity should +still be used on read-only filesystems. fs-verity is for files that +must live on a read-write filesystem because they are independently +updated and potentially user-installed, so dm-verity cannot be used. + +The base fs-verity feature is a hashing mechanism only; actually +authenticating the files is up to userspace. However, to meet some +users' needs, fs-verity optionally supports a simple signature +verification mechanism where users can configure the kernel to require +that all fs-verity files be signed by a key loaded into a keyring; see +`Built-in signature verification`_. Support for fs-verity file hashes +in IMA (Integrity Measurement Architecture) policies is also planned. + +User API +======== + +FS_IOC_ENABLE_VERITY +-------------------- + +The FS_IOC_ENABLE_VERITY ioctl enables fs-verity on a file. It takes +in a pointer to a :c:type:`struct fsverity_enable_arg`, defined as +follows:: + + struct fsverity_enable_arg { + __u32 version; + __u32 hash_algorithm; + __u32 block_size; + __u32 salt_size; + __u64 salt_ptr; + __u32 sig_size; + __u32 __reserved1; + __u64 sig_ptr; + __u64 __reserved2[11]; + }; + +This structure contains the parameters of the Merkle tree to build for +the file, and optionally contains a signature. It must be initialized +as follows: + +- ``version`` must be 1. +- ``hash_algorithm`` must be the identifier for the hash algorithm to + use for the Merkle tree, such as FS_VERITY_HASH_ALG_SHA256. See + ``include/uapi/linux/fsverity.h`` for the list of possible values. +- ``block_size`` must be the Merkle tree block size. Currently, this + must be equal to the system page size, which is usually 4096 bytes. + Other sizes may be supported in the future. This value is not + necessarily the same as the filesystem block size. +- ``salt_size`` is the size of the salt in bytes, or 0 if no salt is + provided. The salt is a value that is prepended to every hashed + block; it can be used to personalize the hashing for a particular + file or device. Currently the maximum salt size is 32 bytes. +- ``salt_ptr`` is the pointer to the salt, or NULL if no salt is + provided. +- ``sig_size`` is the size of the signature in bytes, or 0 if no + signature is provided. Currently the signature is (somewhat + arbitrarily) limited to 16128 bytes. See `Built-in signature + verification`_ for more information. +- ``sig_ptr`` is the pointer to the signature, or NULL if no + signature is provided. +- All reserved fields must be zeroed. + +FS_IOC_ENABLE_VERITY causes the filesystem to build a Merkle tree for +the file and persist it to a filesystem-specific location associated +with the file, then mark the file as a verity file. This ioctl may +take a long time to execute on large files, and it is interruptible by +fatal signals. + +FS_IOC_ENABLE_VERITY checks for write access to the inode. However, +it must be executed on an O_RDONLY file descriptor and no processes +can have the file open for writing. Attempts to open the file for +writing while this ioctl is executing will fail with ETXTBSY. (This +is necessary to guarantee that no writable file descriptors will exist +after verity is enabled, and to guarantee that the file's contents are +stable while the Merkle tree is being built over it.) + +On success, FS_IOC_ENABLE_VERITY returns 0, and the file becomes a +verity file. On failure (including the case of interruption by a +fatal signal), no changes are made to the file. + +FS_IOC_ENABLE_VERITY can fail with the following errors: + +- ``EACCES``: the process does not have write access to the file +- ``EBADMSG``: the signature is malformed +- ``EBUSY``: this ioctl is already running on the file +- ``EEXIST``: the file already has verity enabled +- ``EFAULT``: the caller provided inaccessible memory +- ``EINTR``: the operation was interrupted by a fatal signal +- ``EINVAL``: unsupported version, hash algorithm, or block size; or + reserved bits are set; or the file descriptor refers to neither a + regular file nor a directory. +- ``EISDIR``: the file descriptor refers to a directory +- ``EKEYREJECTED``: the signature doesn't match the file +- ``EMSGSIZE``: the salt or signature is too long +- ``ENOKEY``: the fs-verity keyring doesn't contain the certificate + needed to verify the signature +- ``ENOPKG``: fs-verity recognizes the hash algorithm, but it's not + available in the kernel's crypto API as currently configured (e.g. + for SHA-512, missing CONFIG_CRYPTO_SHA512). +- ``ENOTTY``: this type of filesystem does not implement fs-verity +- ``EOPNOTSUPP``: the kernel was not configured with fs-verity + support; or the filesystem superblock has not had the 'verity' + feature enabled on it; or the filesystem does not support fs-verity + on this file. (See `Filesystem support`_.) +- ``EPERM``: the file is append-only; or, a signature is required and + one was not provided. +- ``EROFS``: the filesystem is read-only +- ``ETXTBSY``: someone has the file open for writing. This can be the + caller's file descriptor, another open file descriptor, or the file + reference held by a writable memory map. + +FS_IOC_MEASURE_VERITY +--------------------- + +The FS_IOC_MEASURE_VERITY ioctl retrieves the measurement of a verity +file. The file measurement is a digest that cryptographically +identifies the file contents that are being enforced on reads. + +This ioctl takes in a pointer to a variable-length structure:: + + struct fsverity_digest { + __u16 digest_algorithm; + __u16 digest_size; /* input/output */ + __u8 digest[]; + }; + +``digest_size`` is an input/output field. On input, it must be +initialized to the number of bytes allocated for the variable-length +``digest`` field. + +On success, 0 is returned and the kernel fills in the structure as +follows: + +- ``digest_algorithm`` will be the hash algorithm used for the file + measurement. It will match ``fsverity_enable_arg::hash_algorithm``. +- ``digest_size`` will be the size of the digest in bytes, e.g. 32 + for SHA-256. (This can be redundant with ``digest_algorithm``.) +- ``digest`` will be the actual bytes of the digest. + +FS_IOC_MEASURE_VERITY is guaranteed to execute in constant time, +regardless of the size of the file. + +FS_IOC_MEASURE_VERITY can fail with the following errors: + +- ``EFAULT``: the caller provided inaccessible memory +- ``ENODATA``: the file is not a verity file +- ``ENOTTY``: this type of filesystem does not implement fs-verity +- ``EOPNOTSUPP``: the kernel was not configured with fs-verity + support, or the filesystem superblock has not had the 'verity' + feature enabled on it. (See `Filesystem support`_.) +- ``EOVERFLOW``: the digest is longer than the specified + ``digest_size`` bytes. Try providing a larger buffer. + +FS_IOC_GETFLAGS +--------------- + +The existing ioctl FS_IOC_GETFLAGS (which isn't specific to fs-verity) +can also be used to check whether a file has fs-verity enabled or not. +To do so, check for FS_VERITY_FL (0x00100000) in the returned flags. + +The verity flag is not settable via FS_IOC_SETFLAGS. You must use +FS_IOC_ENABLE_VERITY instead, since parameters must be provided. + +Accessing verity files +====================== + +Applications can transparently access a verity file just like a +non-verity one, with the following exceptions: + +- Verity files are readonly. They cannot be opened for writing or + truncate()d, even if the file mode bits allow it. Attempts to do + one of these things will fail with EPERM. However, changes to + metadata such as owner, mode, timestamps, and xattrs are still + allowed, since these are not measured by fs-verity. Verity files + can also still be renamed, deleted, and linked to. + +- Direct I/O is not supported on verity files. Attempts to use direct + I/O on such files will fall back to buffered I/O. + +- DAX (Direct Access) is not supported on verity files, because this + would circumvent the data verification. + +- Reads of data that doesn't match the verity Merkle tree will fail + with EIO (for read()) or SIGBUS (for mmap() reads). + +- If the sysctl "fs.verity.require_signatures" is set to 1 and the + file's verity measurement is not signed by a key in the fs-verity + keyring, then opening the file will fail. See `Built-in signature + verification`_. + +Direct access to the Merkle tree is not supported. Therefore, if a +verity file is copied, or is backed up and restored, then it will lose +its "verity"-ness. fs-verity is primarily meant for files like +executables that are managed by a package manager. + +File measurement computation +============================ + +This section describes how fs-verity hashes the file contents using a +Merkle tree to produce the "file measurement" which cryptographically +identifies the file contents. This algorithm is the same for all +filesystems that support fs-verity. + +Userspace only needs to be aware of this algorithm if it needs to +compute the file measurement itself, e.g. in order to sign the file. + +.. _fsverity_merkle_tree: + +Merkle tree +----------- + +The file contents is divided into blocks, where the block size is +configurable but is usually 4096 bytes. The end of the last block is +zero-padded if needed. Each block is then hashed, producing the first +level of hashes. Then, the hashes in this first level are grouped +into 'blocksize'-byte blocks (zero-padding the ends as needed) and +these blocks are hashed, producing the second level of hashes. This +proceeds up the tree until only a single block remains. The hash of +this block is the "Merkle tree root hash". + +If the file fits in one block and is nonempty, then the "Merkle tree +root hash" is simply the hash of the single data block. If the file +is empty, then the "Merkle tree root hash" is all zeroes. + +The "blocks" here are not necessarily the same as "filesystem blocks". + +If a salt was specified, then it's zero-padded to the closest multiple +of the input size of the hash algorithm's compression function, e.g. +64 bytes for SHA-256 or 128 bytes for SHA-512. The padded salt is +prepended to every data or Merkle tree block that is hashed. + +The purpose of the block padding is to cause every hash to be taken +over the same amount of data, which simplifies the implementation and +keeps open more possibilities for hardware acceleration. The purpose +of the salt padding is to make the salting "free" when the salted hash +state is precomputed, then imported for each hash. + +Example: in the recommended configuration of SHA-256 and 4K blocks, +128 hash values fit in each block. Thus, each level of the Merkle +tree is approximately 128 times smaller than the previous, and for +large files the Merkle tree's size converges to approximately 1/127 of +the original file size. However, for small files, the padding is +significant, making the space overhead proportionally more. + +.. _fsverity_descriptor: + +fs-verity descriptor +-------------------- + +By itself, the Merkle tree root hash is ambiguous. For example, it +can't a distinguish a large file from a small second file whose data +is exactly the top-level hash block of the first file. Ambiguities +also arise from the convention of padding to the next block boundary. + +To solve this problem, the verity file measurement is actually +computed as a hash of the following structure, which contains the +Merkle tree root hash as well as other fields such as the file size:: + + struct fsverity_descriptor { + __u8 version; /* must be 1 */ + __u8 hash_algorithm; /* Merkle tree hash algorithm */ + __u8 log_blocksize; /* log2 of size of data and tree blocks */ + __u8 salt_size; /* size of salt in bytes; 0 if none */ + __le32 sig_size; /* must be 0 */ + __le64 data_size; /* size of file the Merkle tree is built over */ + __u8 root_hash[64]; /* Merkle tree root hash */ + __u8 salt[32]; /* salt prepended to each hashed block */ + __u8 __reserved[144]; /* must be 0's */ + }; + +Note that the ``sig_size`` field must be set to 0 for the purpose of +computing the file measurement, even if a signature was provided (or +will be provided) to `FS_IOC_ENABLE_VERITY`_. + +Built-in signature verification +=============================== + +With CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y, fs-verity supports putting +a portion of an authentication policy (see `Use cases`_) in the +kernel. Specifically, it adds support for: + +1. At fs-verity module initialization time, a keyring ".fs-verity" is + created. The root user can add trusted X.509 certificates to this + keyring using the add_key() system call, then (when done) + optionally use keyctl_restrict_keyring() to prevent additional + certificates from being added. + +2. `FS_IOC_ENABLE_VERITY`_ accepts a pointer to a PKCS#7 formatted + detached signature in DER format of the file measurement. On + success, this signature is persisted alongside the Merkle tree. + Then, any time the file is opened, the kernel will verify the + file's actual measurement against this signature, using the + certificates in the ".fs-verity" keyring. + +3. A new sysctl "fs.verity.require_signatures" is made available. + When set to 1, the kernel requires that all verity files have a + correctly signed file measurement as described in (2). + +File measurements must be signed in the following format, which is +similar to the structure used by `FS_IOC_MEASURE_VERITY`_:: + + struct fsverity_signed_digest { + char magic[8]; /* must be "FSVerity" */ + __le16 digest_algorithm; + __le16 digest_size; + __u8 digest[]; + }; + +fs-verity's built-in signature verification support is meant as a +relatively simple mechanism that can be used to provide some level of +authenticity protection for verity files, as an alternative to doing +the signature verification in userspace or using IMA-appraisal. +However, with this mechanism, userspace programs still need to check +that the verity bit is set, and there is no protection against verity +files being swapped around. + +Filesystem support +================== + +fs-verity is currently supported by the ext4 and f2fs filesystems. +The CONFIG_FS_VERITY kconfig option must be enabled to use fs-verity +on either filesystem. + +``include/linux/fsverity.h`` declares the interface between the +``fs/verity/`` support layer and filesystems. Briefly, filesystems +must provide an ``fsverity_operations`` structure that provides +methods to read and write the verity metadata to a filesystem-specific +location, including the Merkle tree blocks and +``fsverity_descriptor``. Filesystems must also call functions in +``fs/verity/`` at certain times, such as when a file is opened or when +pages have been read into the pagecache. (See `Verifying data`_.) + +ext4 +---- + +ext4 supports fs-verity since Linux TODO and e2fsprogs v1.45.2. + +To create verity files on an ext4 filesystem, the filesystem must have +been formatted with ``-O verity`` or had ``tune2fs -O verity`` run on +it. "verity" is an RO_COMPAT filesystem feature, so once set, old +kernels will only be able to mount the filesystem readonly, and old +versions of e2fsck will be unable to check the filesystem. Moreover, +currently ext4 only supports mounting a filesystem with the "verity" +feature when its block size is equal to PAGE_SIZE (often 4096 bytes). + +ext4 sets the EXT4_VERITY_FL on-disk inode flag on verity files. It +can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be cleared. + +ext4 also supports encryption, which can be used simultaneously with +fs-verity. In this case, the plaintext data is verified rather than +the ciphertext. This is necessary in order to make the file +measurement meaningful, since every file is encrypted differently. + +ext4 stores the verity metadata (Merkle tree and fsverity_descriptor) +past the end of the file, starting at the first 64K boundary beyond +i_size. This approach works because (a) verity files are readonly, +and (b) pages fully beyond i_size aren't visible to userspace but can +be read/written internally by ext4 with only some relatively small +changes to ext4. This approach avoids having to depend on the +EA_INODE feature and on rearchitecturing ext4's xattr support to +support paging multi-gigabyte xattrs into memory, and to support +encrypting xattrs. Note that the verity metadata *must* be encrypted +when the file is, since it contains hashes of the plaintext data. + +Currently, ext4 verity only supports the case where the Merkle tree +block size, filesystem block size, and page size are all the same. It +also only supports extent-based files. + +f2fs +---- + +f2fs supports fs-verity since Linux TODO and f2fs-tools v1.11.0. + +To create verity files on an f2fs filesystem, the filesystem must have +been formatted with ``-O verity``. + +f2fs sets the FADVISE_VERITY_BIT on-disk inode flag on verity files. +It can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be +cleared. + +Like ext4, f2fs stores the verity metadata (Merkle tree and +fsverity_descriptor) past the end of the file, starting at the first +64K boundary beyond i_size. See explanation for ext4 above. +Moreover, f2fs supports at most 4096 bytes of xattr entries per inode +which wouldn't be enough for even a single Merkle tree block. + +Currently, f2fs verity only supports a Merkle tree block size of 4096. +Also, f2fs doesn't support enabling verity on files that currently +have atomic or volatile writes pending. + +Implementation details +====================== + +Verifying data +-------------- + +fs-verity ensures that all reads of a verity file's data are verified, +regardless of which syscall is used to do the read (e.g. mmap(), +read(), pread()) and regardless of whether it's the first read or a +later read (unless the later read can return cached data that was +already verified). Below, we describe how filesystems implement this. + +Pagecache +~~~~~~~~~ + +For filesystems using Linux's pagecache, the ``->readpage()`` and +``->readpages()`` methods must be modified to verify pages before they +are marked Uptodate. Merely hooking ``->read_iter()`` would be +insufficient, since ``->read_iter()`` is not used for memory maps. + +Therefore, fs/verity/ provides a function fsverity_verify_page() which +verifies a page that has been read into the pagecache of a verity +inode, but is still locked and not Uptodate, so it's not yet readable +by userspace. As needed to do the verification, +fsverity_verify_page() will call back into the filesystem to read +Merkle tree pages via fsverity_operations::read_merkle_tree_page(). + +fsverity_verify_page() returns false if verification failed; in this +case, the filesystem must not set the page Uptodate. Following this, +as per the usual Linux pagecache behavior, attempts by userspace to +read() from the part of the file containing the page will fail with +EIO, and accesses to the page within a memory map will raise SIGBUS. + +fsverity_verify_page() currently only supports the case where the +Merkle tree block size is equal to PAGE_SIZE (often 4096 bytes). + +In principle, fsverity_verify_page() verifies the entire path in the +Merkle tree from the data page to the root hash. However, for +efficiency the filesystem may cache the hash pages. Therefore, +fsverity_verify_page() only ascends the tree reading hash pages until +an already-verified hash page is seen, as indicated by the PageChecked +bit being set. It then verifies the path to that page. + +This optimization, which is also used by dm-verity, results in +excellent sequential read performance. This is because usually (e.g. +127 in 128 times for 4K blocks and SHA-256) the hash page from the +bottom level of the tree will already be cached and checked from +reading a previous data page. However, random reads perform worse. + +Block device based filesystems +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Block device based filesystems (e.g. ext4 and f2fs) in Linux also use +the pagecache, so the above subsection applies too. However, they +also usually read many pages from a file at once, grouped into a +structure called a "bio". To make it easier for these types of +filesystems to support fs-verity, fs/verity/ also provides a function +fsverity_verify_bio() which verifies all pages in a bio. + +ext4 and f2fs also support encryption. If a verity file is also +encrypted, the pages must be decrypted before being verified. To +support this, these filesystems allocate a "post-read context" for +each bio and store it in ``->bi_private``:: + + struct bio_post_read_ctx { + struct bio *bio; + struct work_struct work; + unsigned int cur_step; + unsigned int enabled_steps; + }; + +``enabled_steps`` is a bitmask that specifies whether decryption, +verity, or both is enabled. After the bio completes, for each needed +postprocessing step the filesystem enqueues the bio_post_read_ctx on a +workqueue, and then the workqueue work does the decryption or +verification. Finally, pages where no decryption or verity error +occurred are marked Uptodate, and the pages are unlocked. + +Files on ext4 and f2fs may contain holes. Normally, ``->readpages()`` +simply zeroes holes and sets the corresponding pages Uptodate; no bios +are issued. To prevent this case from bypassing fs-verity, these +filesystems use fsverity_verify_page() to verify hole pages. + +ext4 and f2fs disable direct I/O on verity files, since otherwise +direct I/O would bypass fs-verity. (They also do the same for +encrypted files.) + +Userspace utility +================= + +This document focuses on the kernel, but a userspace utility for +fs-verity can be found at: + + https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/fsverity-utils.git + +See the README.md file in the fsverity-utils source tree for details, +including examples of setting up fs-verity protected files. + +Tests +===== + +To test fs-verity, use xfstests. For example, using `kvm-xfstests +<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_:: + + kvm-xfstests -c ext4,f2fs -g verity + +FAQ +=== + +This section answers frequently asked questions about fs-verity that +weren't already directly answered in other parts of this document. + +:Q: Why isn't fs-verity part of IMA? +:A: fs-verity and IMA (Integrity Measurement Architecture) have + different focuses. fs-verity is a filesystem-level mechanism for + hashing individual files using a Merkle tree. In contrast, IMA + specifies a system-wide policy that specifies which files are + hashed and what to do with those hashes, such as log them, + authenticate them, or add them to a measurement list. + + IMA is planned to support the fs-verity hashing mechanism as an + alternative to doing full file hashes, for people who want the + performance and security benefits of the Merkle tree based hash. + But it doesn't make sense to force all uses of fs-verity to be + through IMA. As a standalone filesystem feature, fs-verity + already meets many users' needs, and it's testable like other + filesystem features e.g. with xfstests. + +:Q: Isn't fs-verity useless because the attacker can just modify the + hashes in the Merkle tree, which is stored on-disk? +:A: To verify the authenticity of an fs-verity file you must verify + the authenticity of the "file measurement", which is basically the + root hash of the Merkle tree. See `Use cases`_. + +:Q: Isn't fs-verity useless because the attacker can just replace a + verity file with a non-verity one? +:A: See `Use cases`_. In the initial use case, it's really trusted + userspace code that authenticates the files; fs-verity is just a + tool to do this job efficiently and securely. The trusted + userspace code will consider non-verity files to be inauthentic. + +:Q: Why does the Merkle tree need to be stored on-disk? Couldn't you + store just the root hash? +:A: If the Merkle tree wasn't stored on-disk, then you'd have to + compute the entire tree when the file is first accessed, even if + just one byte is being read. This is a fundamental consequence of + how Merkle tree hashing works. To verify a leaf node, you need to + verify the whole path to the root hash, including the root node + (the thing which the root hash is a hash of). But if the root + node isn't stored on-disk, you have to compute it by hashing its + children, and so on until you've actually hashed the entire file. + + That defeats most of the point of doing a Merkle tree-based hash, + since if you have to hash the whole file ahead of time anyway, + then you could simply do sha256(file) instead. That would be much + simpler, and a bit faster too. + + It's true that an in-memory Merkle tree could still provide the + advantage of verification on every read rather than just on the + first read. However, it would be inefficient because every time a + hash page gets evicted (you can't pin the entire Merkle tree into + memory, since it may be very large), in order to restore it you + again need to hash everything below it in the tree. This again + defeats most of the point of doing a Merkle tree-based hash, since + a single block read could trigger re-hashing gigabytes of data. + +:Q: But couldn't you store just the leaf nodes and compute the rest? +:A: See previous answer; this really just moves up one level, since + one could alternatively interpret the data blocks as being the + leaf nodes of the Merkle tree. It's true that the tree can be + computed much faster if the leaf level is stored rather than just + the data, but that's only because each level is less than 1% the + size of the level below (assuming the recommended settings of + SHA-256 and 4K blocks). For the exact same reason, by storing + "just the leaf nodes" you'd already be storing over 99% of the + tree, so you might as well simply store the whole tree. + +:Q: Can the Merkle tree be built ahead of time, e.g. distributed as + part of a package that is installed to many computers? +:A: This isn't currently supported. It was part of the original + design, but was removed to simplify the kernel UAPI and because it + wasn't a critical use case. Files are usually installed once and + used many times, and cryptographic hashing is somewhat fast on + most modern processors. + +:Q: Why doesn't fs-verity support writes? +:A: Write support would be very difficult and would require a + completely different design, so it's well outside the scope of + fs-verity. Write support would require: + + - A way to maintain consistency between the data and hashes, + including all levels of hashes, since corruption after a crash + (especially of potentially the entire file!) is unacceptable. + The main options for solving this are data journalling, + copy-on-write, and log-structured volume. But it's very hard to + retrofit existing filesystems with new consistency mechanisms. + Data journalling is available on ext4, but is very slow. + + - Rebuilding the the Merkle tree after every write, which would be + extremely inefficient. Alternatively, a different authenticated + dictionary structure such as an "authenticated skiplist" could + be used. However, this would be far more complex. + + Compare it to dm-verity vs. dm-integrity. dm-verity is very + simple: the kernel just verifies read-only data against a + read-only Merkle tree. In contrast, dm-integrity supports writes + but is slow, is much more complex, and doesn't actually support + full-device authentication since it authenticates each sector + independently, i.e. there is no "root hash". It doesn't really + make sense for the same device-mapper target to support these two + very different cases; the same applies to fs-verity. + +:Q: Since verity files are immutable, why isn't the immutable bit set? +:A: The existing "immutable" bit (FS_IMMUTABLE_FL) already has a + specific set of semantics which not only make the file contents + read-only, but also prevent the file from being deleted, renamed, + linked to, or having its owner or mode changed. These extra + properties are unwanted for fs-verity, so reusing the immutable + bit isn't appropriate. + +:Q: Why does the API use ioctls instead of setxattr() and getxattr()? +:A: Abusing the xattr interface for basically arbitrary syscalls is + heavily frowned upon by most of the Linux filesystem developers. + An xattr should really just be an xattr on-disk, not an API to + e.g. magically trigger construction of a Merkle tree. + +:Q: Does fs-verity support remote filesystems? +:A: Only ext4 and f2fs support is implemented currently, but in + principle any filesystem that can store per-file verity metadata + can support fs-verity, regardless of whether it's local or remote. + Some filesystems may have fewer options of where to store the + verity metadata; one possibility is to store it past the end of + the file and "hide" it from userspace by manipulating i_size. The + data verification functions provided by ``fs/verity/`` also assume + that the filesystem uses the Linux pagecache, but both local and + remote filesystems normally do so. + +:Q: Why is anything filesystem-specific at all? Shouldn't fs-verity + be implemented entirely at the VFS level? +:A: There are many reasons why this is not possible or would be very + difficult, including the following: + + - To prevent bypassing verification, pages must not be marked + Uptodate until they've been verified. Currently, each + filesystem is responsible for marking pages Uptodate via + ``->readpages()``. Therefore, currently it's not possible for + the VFS to do the verification on its own. Changing this would + require significant changes to the VFS and all filesystems. + + - It would require defining a filesystem-independent way to store + the verity metadata. Extended attributes don't work for this + because (a) the Merkle tree may be gigabytes, but many + filesystems assume that all xattrs fit into a single 4K + filesystem block, and (b) ext4 and f2fs encryption doesn't + encrypt xattrs, yet the Merkle tree *must* be encrypted when the + file contents are, because it stores hashes of the plaintext + file contents. + + So the verity metadata would have to be stored in an actual + file. Using a separate file would be very ugly, since the + metadata is fundamentally part of the file to be protected, and + it could cause problems where users could delete the real file + but not the metadata file or vice versa. On the other hand, + having it be in the same file would break applications unless + filesystems' notion of i_size were divorced from the VFS's, + which would be complex and require changes to all filesystems. + + - It's desirable that FS_IOC_ENABLE_VERITY uses the filesystem's + transaction mechanism so that either the file ends up with + verity enabled, or no changes were made. Allowing intermediate + states to occur after a crash may cause problems. diff --git a/Documentation/filesystems/index.rst b/Documentation/filesystems/index.rst index 1131c34d77f6..2c3a9f761205 100644 --- a/Documentation/filesystems/index.rst +++ b/Documentation/filesystems/index.rst @@ -16,9 +16,14 @@ algorithms work. .. toctree:: :maxdepth: 2 - path-lookup.rst + vfs + path-lookup api-summary splice + locking + directory-locking + + porting Filesystem support layers ========================= @@ -31,13 +36,14 @@ filesystem implementations. journalling fscrypt + fsverity -Filesystem-specific documentation -================================= +Filesystems +=========== -Documentation for individual filesystem types can be found here. +Documentation for filesystem implementations. .. toctree:: :maxdepth: 2 - binderfs.rst + virtiofs diff --git a/Documentation/filesystems/jfs.txt b/Documentation/filesystems/jfs.txt deleted file mode 100644 index 41fd757997b3..000000000000 --- a/Documentation/filesystems/jfs.txt +++ /dev/null @@ -1,52 +0,0 @@ -IBM's Journaled File System (JFS) for Linux - -JFS Homepage: http://jfs.sourceforge.net/ - -The following mount options are supported: -(*) == default - -iocharset=name Character set to use for converting from Unicode to - ASCII. The default is to do no conversion. Use - iocharset=utf8 for UTF-8 translations. This requires - CONFIG_NLS_UTF8 to be set in the kernel .config file. - iocharset=none specifies the default behavior explicitly. - -resize=value Resize the volume to <value> blocks. JFS only supports - growing a volume, not shrinking it. This option is only - valid during a remount, when the volume is mounted - read-write. The resize keyword with no value will grow - the volume to the full size of the partition. - -nointegrity Do not write to the journal. The primary use of this option - is to allow for higher performance when restoring a volume - from backup media. The integrity of the volume is not - guaranteed if the system abnormally abends. - -integrity(*) Commit metadata changes to the journal. Use this option to - remount a volume where the nointegrity option was - previously specified in order to restore normal behavior. - -errors=continue Keep going on a filesystem error. -errors=remount-ro(*) Remount the filesystem read-only on an error. -errors=panic Panic and halt the machine if an error occurs. - -uid=value Override on-disk uid with specified value -gid=value Override on-disk gid with specified value -umask=value Override on-disk umask with specified octal value. For - directories, the execute bit will be set if the corresponding - read bit is set. - -discard=minlen This enables/disables the use of discard/TRIM commands. -discard The discard/TRIM commands are sent to the underlying -nodiscard(*) block device when blocks are freed. This is useful for SSD - devices and sparse/thinly-provisioned LUNs. The FITRIM ioctl - command is also available together with the nodiscard option. - The value of minlen specifies the minimum blockcount, when - a TRIM command to the block device is considered useful. - When no value is given to the discard option, it defaults to - 64 blocks, which means 256KiB in JFS. - The minlen value of discard overrides the minlen value given - on an FITRIM ioctl(). - -The JFS mailing list can be subscribed to by using the link labeled -"Mail list Subscribe" at our web page http://jfs.sourceforge.net/ diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/locking.rst index dac435575384..fc3a0704553c 100644 --- a/Documentation/filesystems/Locking +++ b/Documentation/filesystems/locking.rst @@ -1,14 +1,22 @@ - The text below describes the locking rules for VFS-related methods. +======= +Locking +======= + +The text below describes the locking rules for VFS-related methods. It is (believed to be) up-to-date. *Please*, if you change anything in prototypes or locking protocols - update this file. And update the relevant instances in the tree, don't leave that to maintainers of filesystems/devices/ etc. At the very least, put the list of dubious cases in the end of this file. Don't turn it into log - maintainers of out-of-the-tree code are supposed to be able to use diff(1). - Thing currently missing here: socket operations. Alexey? ---------------------------- dentry_operations -------------------------- -prototypes: +Thing currently missing here: socket operations. Alexey? + +dentry_operations +================= + +prototypes:: + int (*d_revalidate)(struct dentry *, unsigned int); int (*d_weak_revalidate)(struct dentry *, unsigned int); int (*d_hash)(const struct dentry *, struct qstr *); @@ -24,23 +32,30 @@ prototypes: struct dentry *(*d_real)(struct dentry *, const struct inode *); locking rules: - rename_lock ->d_lock may block rcu-walk -d_revalidate: no no yes (ref-walk) maybe -d_weak_revalidate:no no yes no -d_hash no no no maybe -d_compare: yes no no maybe -d_delete: no yes no no -d_init: no no yes no -d_release: no no yes no -d_prune: no yes no no -d_iput: no no yes no -d_dname: no no no no -d_automount: no no yes no -d_manage: no no yes (ref-walk) maybe -d_real no no yes no - ---------------------------- inode_operations --------------------------- -prototypes: + +================== =========== ======== ============== ======== +ops rename_lock ->d_lock may block rcu-walk +================== =========== ======== ============== ======== +d_revalidate: no no yes (ref-walk) maybe +d_weak_revalidate: no no yes no +d_hash no no no maybe +d_compare: yes no no maybe +d_delete: no yes no no +d_init: no no yes no +d_release: no no yes no +d_prune: no yes no no +d_iput: no no yes no +d_dname: no no no no +d_automount: no no yes no +d_manage: no no yes (ref-walk) maybe +d_real no no yes no +================== =========== ======== ============== ======== + +inode_operations +================ + +prototypes:: + int (*create) (struct inode *,struct dentry *,umode_t, bool); struct dentry * (*lookup) (struct inode *,struct dentry *, unsigned int); int (*link) (struct dentry *,struct inode *,struct dentry *); @@ -68,7 +83,10 @@ prototypes: locking rules: all may block - i_rwsem(inode) + +============ ============================================= +ops i_rwsem(inode) +============ ============================================= lookup: shared create: exclusive link: exclusive (both) @@ -89,17 +107,21 @@ fiemap: no update_time: no atomic_open: exclusive tmpfile: no +============ ============================================= Additionally, ->rmdir(), ->unlink() and ->rename() have ->i_rwsem exclusive on victim. cross-directory ->rename() has (per-superblock) ->s_vfs_rename_sem. -See Documentation/filesystems/directory-locking for more detailed discussion +See Documentation/filesystems/directory-locking.rst for more detailed discussion of the locking scheme for directory operations. ------------------------ xattr_handler operations ----------------------- -prototypes: +xattr_handler operations +======================== + +prototypes:: + bool (*list)(struct dentry *dentry); int (*get)(const struct xattr_handler *handler, struct dentry *dentry, struct inode *inode, const char *name, void *buffer, @@ -110,13 +132,20 @@ prototypes: locking rules: all may block - i_rwsem(inode) + +===== ============== +ops i_rwsem(inode) +===== ============== list: no get: no set: exclusive +===== ============== + +super_operations +================ + +prototypes:: ---------------------------- super_operations --------------------------- -prototypes: struct inode *(*alloc_inode)(struct super_block *sb); void (*free_inode)(struct inode *); void (*destroy_inode)(struct inode *); @@ -138,7 +167,10 @@ prototypes: locking rules: All may block [not true, see below] - s_umount + +====================== ============ ======================== +ops s_umount note +====================== ============ ======================== alloc_inode: free_inode: called from RCU callback destroy_inode: @@ -157,6 +189,7 @@ show_options: no (namespace_sem) quota_read: no (see below) quota_write: no (see below) bdev_try_to_free_page: no (see below) +====================== ============ ======================== ->statfs() has s_umount (shared) when called by ustat(2) (native or compat), but that's an accident of bad API; s_umount is used to pin @@ -164,31 +197,44 @@ the superblock down when we only have dev_t given us by userland to identify the superblock. Everything else (statfs(), fstatfs(), etc.) doesn't hold it when calling ->statfs() - superblock is pinned down by resolving the pathname passed to syscall. + ->quota_read() and ->quota_write() functions are both guaranteed to be the only ones operating on the quota file by the quota code (via dqio_sem) (unless an admin really wants to screw up something and writes to quota files with quotas on). For other details about locking see also dquot_operations section. + ->bdev_try_to_free_page is called from the ->releasepage handler of the block device inode. See there for more details. ---------------------------- file_system_type --------------------------- -prototypes: +file_system_type +================ + +prototypes:: + struct dentry *(*mount) (struct file_system_type *, int, const char *, void *); void (*kill_sb) (struct super_block *); + locking rules: - may block + +======= ========= +ops may block +======= ========= mount yes kill_sb yes +======= ========= ->mount() returns ERR_PTR or the root dentry; its superblock should be locked on return. + ->kill_sb() takes a write-locked superblock, does all shutdown work on it, unlocks and drops the reference. ---------------------------- address_space_operations -------------------------- -prototypes: +address_space_operations +======================== +prototypes:: + int (*writepage)(struct page *page, struct writeback_control *wbc); int (*readpage)(struct file *, struct page *); int (*writepages)(struct address_space *, struct writeback_control *); @@ -218,14 +264,16 @@ prototypes: locking rules: All except set_page_dirty and freepage may block - PageLocked(page) i_rwsem +====================== ======================== ========= +ops PageLocked(page) i_rwsem +====================== ======================== ========= writepage: yes, unlocks (see below) readpage: yes, unlocks writepages: set_page_dirty no readpages: -write_begin: locks the page exclusive -write_end: yes, unlocks exclusive +write_begin: locks the page exclusive +write_end: yes, unlocks exclusive bmap: invalidatepage: yes releasepage: yes @@ -239,17 +287,18 @@ is_partially_uptodate: yes error_remove_page: yes swap_activate: no swap_deactivate: no +====================== ======================== ========= - ->write_begin(), ->write_end() and ->readpage() may be called from +->write_begin(), ->write_end() and ->readpage() may be called from the request handler (/dev/loop). - ->readpage() unlocks the page, either synchronously or via I/O +->readpage() unlocks the page, either synchronously or via I/O completion. - ->readpages() populates the pagecache with the passed pages and starts +->readpages() populates the pagecache with the passed pages and starts I/O against them. They come unlocked upon I/O completion. - ->writepage() is used for two purposes: for "memory cleansing" and for +->writepage() is used for two purposes: for "memory cleansing" and for "sync". These are quite different operations and the behaviour may differ depending upon the mode. @@ -297,72 +346,81 @@ will leave the page itself marked clean but it will be tagged as dirty in the radix tree. This incoherency can lead to all sorts of hard-to-debug problems in the filesystem like having dirty inodes at umount and losing written data. - ->writepages() is used for periodic writeback and for syscall-initiated +->writepages() is used for periodic writeback and for syscall-initiated sync operations. The address_space should start I/O against at least -*nr_to_write pages. *nr_to_write must be decremented for each page which is -written. The address_space implementation may write more (or less) pages -than *nr_to_write asks for, but it should try to be reasonably close. If -nr_to_write is NULL, all dirty pages must be written. +``*nr_to_write`` pages. ``*nr_to_write`` must be decremented for each page +which is written. The address_space implementation may write more (or less) +pages than ``*nr_to_write`` asks for, but it should try to be reasonably close. +If nr_to_write is NULL, all dirty pages must be written. writepages should _only_ write pages which are present on mapping->io_pages. - ->set_page_dirty() is called from various places in the kernel +->set_page_dirty() is called from various places in the kernel when the target page is marked as needing writeback. It may be called under spinlock (it cannot block) and is sometimes called with the page not locked. - ->bmap() is currently used by legacy ioctl() (FIBMAP) provided by some +->bmap() is currently used by legacy ioctl() (FIBMAP) provided by some filesystems and by the swapper. The latter will eventually go away. Please, keep it that way and don't breed new callers. - ->invalidatepage() is called when the filesystem must attempt to drop +->invalidatepage() is called when the filesystem must attempt to drop some or all of the buffers from the page when it is being truncated. It returns zero on success. If ->invalidatepage is zero, the kernel uses block_invalidatepage() instead. - ->releasepage() is called when the kernel is about to try to drop the +->releasepage() is called when the kernel is about to try to drop the buffers from the page in preparation for freeing it. It returns zero to indicate that the buffers are (or may be) freeable. If ->releasepage is zero, the kernel assumes that the fs has no private interest in the buffers. - ->freepage() is called when the kernel is done dropping the page +->freepage() is called when the kernel is done dropping the page from the page cache. - ->launder_page() may be called prior to releasing a page if +->launder_page() may be called prior to releasing a page if it is still found to be dirty. It returns zero if the page was successfully cleaned, or an error value if not. Note that in order to prevent the page getting mapped back in and redirtied, it needs to be kept locked across the entire operation. - ->swap_activate will be called with a non-zero argument on +->swap_activate will be called with a non-zero argument on files backing (non block device backed) swapfiles. A return value of zero indicates success, in which case this file can be used for backing swapspace. The swapspace operations will be proxied to the address space operations. - ->swap_deactivate() will be called in the sys_swapoff() +->swap_deactivate() will be called in the sys_swapoff() path after ->swap_activate() returned success. ------------------------ file_lock_operations ------------------------------ -prototypes: +file_lock_operations +==================== + +prototypes:: + void (*fl_copy_lock)(struct file_lock *, struct file_lock *); void (*fl_release_private)(struct file_lock *); locking rules: - inode->i_lock may block + +=================== ============= ========= +ops inode->i_lock may block +=================== ============= ========= fl_copy_lock: yes no -fl_release_private: maybe maybe[1] +fl_release_private: maybe maybe[1]_ +=================== ============= ========= + +.. [1]: + ->fl_release_private for flock or POSIX locks is currently allowed + to block. Leases however can still be freed while the i_lock is held and + so fl_release_private called on a lease should not block. -[1]: ->fl_release_private for flock or POSIX locks is currently allowed -to block. Leases however can still be freed while the i_lock is held and -so fl_release_private called on a lease should not block. +lock_manager_operations +======================= + +prototypes:: ------------------------ lock_manager_operations --------------------------- -prototypes: - int (*lm_compare_owner)(struct file_lock *, struct file_lock *); - unsigned long (*lm_owner_key)(struct file_lock *); void (*lm_notify)(struct file_lock *); /* unblock callback */ int (*lm_grant)(struct file_lock *, struct file_lock *, int); void (*lm_break)(struct file_lock *); /* break_lease callback */ @@ -370,36 +428,33 @@ prototypes: locking rules: - inode->i_lock blocked_lock_lock may block -lm_compare_owner: yes[1] maybe no -lm_owner_key yes[1] yes no +========== ============= ================= ========= +ops inode->i_lock blocked_lock_lock may block +========== ============= ================= ========= lm_notify: yes yes no lm_grant: no no no lm_break: yes no no lm_change yes no no +========== ============= ================= ========= + +buffer_head +=========== + +prototypes:: -[1]: ->lm_compare_owner and ->lm_owner_key are generally called with -*an* inode->i_lock held. It may not be the i_lock of the inode -associated with either file_lock argument! This is the case with deadlock -detection, since the code has to chase down the owners of locks that may -be entirely unrelated to the one on which the lock is being acquired. -For deadlock detection however, the blocked_lock_lock is also held. The -fact that these locks are held ensures that the file_locks do not -disappear out from under you while doing the comparison or generating an -owner key. - ---------------------------- buffer_head ----------------------------------- -prototypes: void (*b_end_io)(struct buffer_head *bh, int uptodate); locking rules: - called from interrupts. In other words, extreme care is needed here. + +called from interrupts. In other words, extreme care is needed here. bh is locked, but that's all warranties we have here. Currently only RAID1, highmem, fs/buffer.c, and fs/ntfs/aops.c are providing these. Block devices call this method upon the IO completion. ---------------------------- block_device_operations ----------------------- -prototypes: +block_device_operations +======================= +prototypes:: + int (*open) (struct block_device *, fmode_t); int (*release) (struct gendisk *, fmode_t); int (*ioctl) (struct block_device *, fmode_t, unsigned, unsigned long); @@ -413,7 +468,10 @@ prototypes: void (*swap_slot_free_notify) (struct block_device *, unsigned long); locking rules: - bd_mutex + +======================= =================== +ops bd_mutex +======================= =================== open: yes release: yes ioctl: no @@ -424,6 +482,7 @@ unlock_native_capacity: no revalidate_disk: no getgeo: no swap_slot_free_notify: no (see below) +======================= =================== media_changed, unlock_native_capacity and revalidate_disk are called only from check_disk_change(). @@ -432,8 +491,11 @@ swap_slot_free_notify is called with swap_lock and sometimes the page lock held. ---------------------------- file_operations ------------------------------- -prototypes: +file_operations +=============== + +prototypes:: + loff_t (*llseek) (struct file *, loff_t, int); ssize_t (*read) (struct file *, char __user *, size_t, loff_t *); ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); @@ -469,7 +531,6 @@ prototypes: size_t, unsigned int); int (*setlease)(struct file *, long, struct file_lock **, void **); long (*fallocate)(struct file *, int, loff_t, loff_t); -}; locking rules: All may block. @@ -504,8 +565,11 @@ in sys_read() and friends. the lease within the individual filesystem to record the result of the operation ---------------------------- dquot_operations ------------------------------- -prototypes: +dquot_operations +================ + +prototypes:: + int (*write_dquot) (struct dquot *); int (*acquire_dquot) (struct dquot *); int (*release_dquot) (struct dquot *); @@ -517,20 +581,26 @@ a proper locking wrt the filesystem and call the generic quota operations. What filesystem should expect from the generic quota functions: - FS recursion Held locks when called +============== ============ ========================= +ops FS recursion Held locks when called +============== ============ ========================= write_dquot: yes dqonoff_sem or dqptr_sem acquire_dquot: yes dqonoff_sem or dqptr_sem release_dquot: yes dqonoff_sem or dqptr_sem mark_dirty: no - write_info: yes dqonoff_sem +============== ============ ========================= FS recursion means calling ->quota_read() and ->quota_write() from superblock operations. More details about quota locking can be found in fs/dquot.c. ---------------------------- vm_operations_struct ----------------------------- -prototypes: +vm_operations_struct +==================== + +prototypes:: + void (*open)(struct vm_area_struct*); void (*close)(struct vm_area_struct*); vm_fault_t (*fault)(struct vm_area_struct*, struct vm_fault *); @@ -539,7 +609,10 @@ prototypes: int (*access)(struct vm_area_struct *, unsigned long, void*, int, int); locking rules: - mmap_sem PageLocked(page) + +============= ======== =========================== +ops mmap_sem PageLocked(page) +============= ======== =========================== open: yes close: yes fault: yes can return with page locked @@ -547,8 +620,9 @@ map_pages: yes page_mkwrite: yes can return with page locked pfn_mkwrite: yes access: yes +============= ======== =========================== - ->fault() is called when a previously not present pte is about +->fault() is called when a previously not present pte is about to be faulted in. The filesystem must find and return the page associated with the passed in "pgoff" in the vm_fault structure. If it is possible that the page may be truncated and/or invalidated, then the filesystem must lock @@ -556,7 +630,7 @@ the page, then ensure it is not already truncated (the page lock will block subsequent truncate), and then return with VM_FAULT_LOCKED, and the page locked. The VM will unlock the page. - ->map_pages() is called when VM asks to map easy accessible pages. +->map_pages() is called when VM asks to map easy accessible pages. Filesystem should find and map pages associated with offsets from "start_pgoff" till "end_pgoff". ->map_pages() is called with page table locked and must not block. If it's not possible to reach a page without blocking, @@ -565,25 +639,26 @@ page table entry. Pointer to entry associated with the page is passed in "pte" field in vm_fault structure. Pointers to entries for other offsets should be calculated relative to "pte". - ->page_mkwrite() is called when a previously read-only pte is +->page_mkwrite() is called when a previously read-only pte is about to become writeable. The filesystem again must ensure that there are no truncate/invalidate races, and then return with the page locked. If the page has been truncated, the filesystem should not look up a new page like the ->fault() handler, but simply return with VM_FAULT_NOPAGE, which will cause the VM to retry the fault. - ->pfn_mkwrite() is the same as page_mkwrite but when the pte is +->pfn_mkwrite() is the same as page_mkwrite but when the pte is VM_PFNMAP or VM_MIXEDMAP with a page-less entry. Expected return is VM_FAULT_NOPAGE. Or one of the VM_FAULT_ERROR types. The default behavior after this call is to make the pte read-write, unless pfn_mkwrite returns an error. - ->access() is called when get_user_pages() fails in +->access() is called when get_user_pages() fails in access_process_vm(), typically used to debug a process through /proc/pid/mem or ptrace. This function is needed only for VM_IO | VM_PFNMAP VMAs. -================================================================================ +-------------------------------------------------------------------------------- + Dubious stuff (if you break something or notice that it is broken and do not fix it yourself diff --git a/Documentation/filesystems/mandatory-locking.txt b/Documentation/filesystems/mandatory-locking.txt index 0979d1d2ca8b..a251ca33164a 100644 --- a/Documentation/filesystems/mandatory-locking.txt +++ b/Documentation/filesystems/mandatory-locking.txt @@ -169,3 +169,13 @@ havoc if they lock crucial files. The way around it is to change the file permissions (remove the setgid bit) before trying to read or write to it. Of course, that might be a bit tricky if the system is hung :-( +7. The "mand" mount option +-------------------------- +Mandatory locking is disabled on all filesystems by default, and must be +administratively enabled by mounting with "-o mand". That mount option +is only allowed if the mounting task has the CAP_SYS_ADMIN capability. + +Since kernel v4.5, it is possible to disable mandatory locking +altogether by setting CONFIG_MANDATORY_FILE_LOCKING to "n". A kernel +with this disabled will reject attempts to mount filesystems with the +"mand" mount option with the error status EPERM. diff --git a/Documentation/filesystems/nfs/Exporting b/Documentation/filesystems/nfs/exporting.rst index 63889149f532..33d588a01ace 100644 --- a/Documentation/filesystems/nfs/Exporting +++ b/Documentation/filesystems/nfs/exporting.rst @@ -1,3 +1,4 @@ +:orphan: Making Filesystems Exportable ============================= @@ -42,9 +43,9 @@ filehandle fragment, there is no automatic creation of a path prefix for the object. This leads to two related but distinct features of the dcache that are not needed for normal filesystem access. -1/ The dcache must sometimes contain objects that are not part of the +1. The dcache must sometimes contain objects that are not part of the proper prefix. i.e that are not connected to the root. -2/ The dcache must be prepared for a newly found (via ->lookup) directory +2. The dcache must be prepared for a newly found (via ->lookup) directory to already have a (non-connected) dentry, and must be able to move that dentry into place (based on the parent and name in the ->lookup). This is particularly needed for directories as @@ -52,7 +53,7 @@ the dcache that are not needed for normal filesystem access. To implement these features, the dcache has: -a/ A dentry flag DCACHE_DISCONNECTED which is set on +a. A dentry flag DCACHE_DISCONNECTED which is set on any dentry that might not be part of the proper prefix. This is set when anonymous dentries are created, and cleared when a dentry is noticed to be a child of a dentry which is in the proper @@ -71,48 +72,52 @@ a/ A dentry flag DCACHE_DISCONNECTED which is set on dentries. That guarantees that we won't need to hunt them down upon umount. -b/ A primitive for creation of secondary roots - d_obtain_root(inode). +b. A primitive for creation of secondary roots - d_obtain_root(inode). Those do _not_ bear DCACHE_DISCONNECTED. They are placed on the per-superblock list (->s_roots), so they can be located at umount time for eviction purposes. -c/ Helper routines to allocate anonymous dentries, and to help attach +c. Helper routines to allocate anonymous dentries, and to help attach loose directory dentries at lookup time. They are: + d_obtain_alias(inode) will return a dentry for the given inode. If the inode already has a dentry, one of those is returned. + If it doesn't, a new anonymous (IS_ROOT and - DCACHE_DISCONNECTED) dentry is allocated and attached. + DCACHE_DISCONNECTED) dentry is allocated and attached. + In the case of a directory, care is taken that only one dentry can ever be attached. + d_splice_alias(inode, dentry) will introduce a new dentry into the tree; either the passed-in dentry or a preexisting alias for the given inode (such as an anonymous one created by d_obtain_alias), if appropriate. It returns NULL when the passed-in dentry is used, following the calling convention of ->lookup. - + Filesystem Issues ----------------- For a filesystem to be exportable it must: - - 1/ provide the filehandle fragment routines described below. - 2/ make sure that d_splice_alias is used rather than d_add + + 1. provide the filehandle fragment routines described below. + 2. make sure that d_splice_alias is used rather than d_add when ->lookup finds an inode for a given parent and name. - If inode is NULL, d_splice_alias(inode, dentry) is equivalent to + If inode is NULL, d_splice_alias(inode, dentry) is equivalent to:: d_add(dentry, inode), NULL Similarly, d_splice_alias(ERR_PTR(err), dentry) = ERR_PTR(err) - Typically the ->lookup routine will simply end with a: + Typically the ->lookup routine will simply end with a:: return d_splice_alias(inode, dentry); } - A file system implementation declares that instances of the filesystem +A file system implementation declares that instances of the filesystem are exportable by setting the s_export_op field in the struct super_block. This field must point to a "struct export_operations" struct which has the following members: diff --git a/Documentation/filesystems/nfs/nfsroot.txt b/Documentation/filesystems/nfs/nfsroot.txt index d2963123eb1c..ae4332464560 100644 --- a/Documentation/filesystems/nfs/nfsroot.txt +++ b/Documentation/filesystems/nfs/nfsroot.txt @@ -239,7 +239,7 @@ rdinit=<executable file> A description of the process of mounting the root file system can be found in: - Documentation/early-userspace/README + Documentation/driver-api/early-userspace/early_userspace_support.rst diff --git a/Documentation/filesystems/overlayfs.txt b/Documentation/filesystems/overlayfs.txt index 1da2f1668f08..845d689e0fd7 100644 --- a/Documentation/filesystems/overlayfs.txt +++ b/Documentation/filesystems/overlayfs.txt @@ -302,7 +302,7 @@ beneath or above the path of another overlay lower layer path. Using an upper layer path and/or a workdir path that are already used by another overlay mount is not allowed and may fail with EBUSY. Using -partially overlapping paths is not allowed but will not fail with EBUSY. +partially overlapping paths is not allowed and may fail with EBUSY. If files are accessed from two overlayfs mounts which share or overlap the upper layer and/or workdir path the behavior of the overlay is undefined, though it will not result in a crash or deadlock. diff --git a/Documentation/filesystems/porting b/Documentation/filesystems/porting deleted file mode 100644 index 3bd1148d8bb6..000000000000 --- a/Documentation/filesystems/porting +++ /dev/null @@ -1,675 +0,0 @@ -Changes since 2.5.0: - ---- -[recommended] - -New helpers: sb_bread(), sb_getblk(), sb_find_get_block(), set_bh(), - sb_set_blocksize() and sb_min_blocksize(). - -Use them. - -(sb_find_get_block() replaces 2.4's get_hash_table()) - ---- -[recommended] - -New methods: ->alloc_inode() and ->destroy_inode(). - -Remove inode->u.foo_inode_i -Declare - struct foo_inode_info { - /* fs-private stuff */ - struct inode vfs_inode; - }; - static inline struct foo_inode_info *FOO_I(struct inode *inode) - { - return list_entry(inode, struct foo_inode_info, vfs_inode); - } - -Use FOO_I(inode) instead of &inode->u.foo_inode_i; - -Add foo_alloc_inode() and foo_destroy_inode() - the former should allocate -foo_inode_info and return the address of ->vfs_inode, the latter should free -FOO_I(inode) (see in-tree filesystems for examples). - -Make them ->alloc_inode and ->destroy_inode in your super_operations. - -Keep in mind that now you need explicit initialization of private data -typically between calling iget_locked() and unlocking the inode. - -At some point that will become mandatory. - ---- -[mandatory] - -Change of file_system_type method (->read_super to ->get_sb) - -->read_super() is no more. Ditto for DECLARE_FSTYPE and DECLARE_FSTYPE_DEV. - -Turn your foo_read_super() into a function that would return 0 in case of -success and negative number in case of error (-EINVAL unless you have more -informative error value to report). Call it foo_fill_super(). Now declare - -int foo_get_sb(struct file_system_type *fs_type, - int flags, const char *dev_name, void *data, struct vfsmount *mnt) -{ - return get_sb_bdev(fs_type, flags, dev_name, data, foo_fill_super, - mnt); -} - -(or similar with s/bdev/nodev/ or s/bdev/single/, depending on the kind of -filesystem). - -Replace DECLARE_FSTYPE... with explicit initializer and have ->get_sb set as -foo_get_sb. - ---- -[mandatory] - -Locking change: ->s_vfs_rename_sem is taken only by cross-directory renames. -Most likely there is no need to change anything, but if you relied on -global exclusion between renames for some internal purpose - you need to -change your internal locking. Otherwise exclusion warranties remain the -same (i.e. parents and victim are locked, etc.). - ---- -[informational] - -Now we have the exclusion between ->lookup() and directory removal (by -->rmdir() and ->rename()). If you used to need that exclusion and do -it by internal locking (most of filesystems couldn't care less) - you -can relax your locking. - ---- -[mandatory] - -->lookup(), ->truncate(), ->create(), ->unlink(), ->mknod(), ->mkdir(), -->rmdir(), ->link(), ->lseek(), ->symlink(), ->rename() -and ->readdir() are called without BKL now. Grab it on entry, drop upon return -- that will guarantee the same locking you used to have. If your method or its -parts do not need BKL - better yet, now you can shift lock_kernel() and -unlock_kernel() so that they would protect exactly what needs to be -protected. - ---- -[mandatory] - -BKL is also moved from around sb operations. BKL should have been shifted into -individual fs sb_op functions. If you don't need it, remove it. - ---- -[informational] - -check for ->link() target not being a directory is done by callers. Feel -free to drop it... - ---- -[informational] - -->link() callers hold ->i_mutex on the object we are linking to. Some of your -problems might be over... - ---- -[mandatory] - -new file_system_type method - kill_sb(superblock). If you are converting -an existing filesystem, set it according to ->fs_flags: - FS_REQUIRES_DEV - kill_block_super - FS_LITTER - kill_litter_super - neither - kill_anon_super -FS_LITTER is gone - just remove it from fs_flags. - ---- -[mandatory] - - FS_SINGLE is gone (actually, that had happened back when ->get_sb() -went in - and hadn't been documented ;-/). Just remove it from fs_flags -(and see ->get_sb() entry for other actions). - ---- -[mandatory] - -->setattr() is called without BKL now. Caller _always_ holds ->i_mutex, so -watch for ->i_mutex-grabbing code that might be used by your ->setattr(). -Callers of notify_change() need ->i_mutex now. - ---- -[recommended] - -New super_block field "struct export_operations *s_export_op" for -explicit support for exporting, e.g. via NFS. The structure is fully -documented at its declaration in include/linux/fs.h, and in -Documentation/filesystems/nfs/Exporting. - -Briefly it allows for the definition of decode_fh and encode_fh operations -to encode and decode filehandles, and allows the filesystem to use -a standard helper function for decode_fh, and provide file-system specific -support for this helper, particularly get_parent. - -It is planned that this will be required for exporting once the code -settles down a bit. - -[mandatory] - -s_export_op is now required for exporting a filesystem. -isofs, ext2, ext3, resierfs, fat -can be used as examples of very different filesystems. - ---- -[mandatory] - -iget4() and the read_inode2 callback have been superseded by iget5_locked() -which has the following prototype, - - struct inode *iget5_locked(struct super_block *sb, unsigned long ino, - int (*test)(struct inode *, void *), - int (*set)(struct inode *, void *), - void *data); - -'test' is an additional function that can be used when the inode -number is not sufficient to identify the actual file object. 'set' -should be a non-blocking function that initializes those parts of a -newly created inode to allow the test function to succeed. 'data' is -passed as an opaque value to both test and set functions. - -When the inode has been created by iget5_locked(), it will be returned with the -I_NEW flag set and will still be locked. The filesystem then needs to finalize -the initialization. Once the inode is initialized it must be unlocked by -calling unlock_new_inode(). - -The filesystem is responsible for setting (and possibly testing) i_ino -when appropriate. There is also a simpler iget_locked function that -just takes the superblock and inode number as arguments and does the -test and set for you. - -e.g. - inode = iget_locked(sb, ino); - if (inode->i_state & I_NEW) { - err = read_inode_from_disk(inode); - if (err < 0) { - iget_failed(inode); - return err; - } - unlock_new_inode(inode); - } - -Note that if the process of setting up a new inode fails, then iget_failed() -should be called on the inode to render it dead, and an appropriate error -should be passed back to the caller. - ---- -[recommended] - -->getattr() finally getting used. See instances in nfs, minix, etc. - ---- -[mandatory] - -->revalidate() is gone. If your filesystem had it - provide ->getattr() -and let it call whatever you had as ->revlidate() + (for symlinks that -had ->revalidate()) add calls in ->follow_link()/->readlink(). - ---- -[mandatory] - -->d_parent changes are not protected by BKL anymore. Read access is safe -if at least one of the following is true: - * filesystem has no cross-directory rename() - * we know that parent had been locked (e.g. we are looking at -->d_parent of ->lookup() argument). - * we are called from ->rename(). - * the child's ->d_lock is held -Audit your code and add locking if needed. Notice that any place that is -not protected by the conditions above is risky even in the old tree - you -had been relying on BKL and that's prone to screwups. Old tree had quite -a few holes of that kind - unprotected access to ->d_parent leading to -anything from oops to silent memory corruption. - ---- -[mandatory] - - FS_NOMOUNT is gone. If you use it - just set SB_NOUSER in flags -(see rootfs for one kind of solution and bdev/socket/pipe for another). - ---- -[recommended] - - Use bdev_read_only(bdev) instead of is_read_only(kdev). The latter -is still alive, but only because of the mess in drivers/s390/block/dasd.c. -As soon as it gets fixed is_read_only() will die. - ---- -[mandatory] - -->permission() is called without BKL now. Grab it on entry, drop upon -return - that will guarantee the same locking you used to have. If -your method or its parts do not need BKL - better yet, now you can -shift lock_kernel() and unlock_kernel() so that they would protect -exactly what needs to be protected. - ---- -[mandatory] - -->statfs() is now called without BKL held. BKL should have been -shifted into individual fs sb_op functions where it's not clear that -it's safe to remove it. If you don't need it, remove it. - ---- -[mandatory] - - is_read_only() is gone; use bdev_read_only() instead. - ---- -[mandatory] - - destroy_buffers() is gone; use invalidate_bdev(). - ---- -[mandatory] - - fsync_dev() is gone; use fsync_bdev(). NOTE: lvm breakage is -deliberate; as soon as struct block_device * is propagated in a reasonable -way by that code fixing will become trivial; until then nothing can be -done. - -[mandatory] - - block truncatation on error exit from ->write_begin, and ->direct_IO -moved from generic methods (block_write_begin, cont_write_begin, -nobh_write_begin, blockdev_direct_IO*) to callers. Take a look at -ext2_write_failed and callers for an example. - -[mandatory] - - ->truncate is gone. The whole truncate sequence needs to be -implemented in ->setattr, which is now mandatory for filesystems -implementing on-disk size changes. Start with a copy of the old inode_setattr -and vmtruncate, and the reorder the vmtruncate + foofs_vmtruncate sequence to -be in order of zeroing blocks using block_truncate_page or similar helpers, -size update and on finally on-disk truncation which should not fail. -setattr_prepare (which used to be inode_change_ok) now includes the size checks -for ATTR_SIZE and must be called in the beginning of ->setattr unconditionally. - -[mandatory] - - ->clear_inode() and ->delete_inode() are gone; ->evict_inode() should -be used instead. It gets called whenever the inode is evicted, whether it has -remaining links or not. Caller does *not* evict the pagecache or inode-associated -metadata buffers; the method has to use truncate_inode_pages_final() to get rid -of those. Caller makes sure async writeback cannot be running for the inode while -(or after) ->evict_inode() is called. - - ->drop_inode() returns int now; it's called on final iput() with -inode->i_lock held and it returns true if filesystems wants the inode to be -dropped. As before, generic_drop_inode() is still the default and it's been -updated appropriately. generic_delete_inode() is also alive and it consists -simply of return 1. Note that all actual eviction work is done by caller after -->drop_inode() returns. - - As before, clear_inode() must be called exactly once on each call of -->evict_inode() (as it used to be for each call of ->delete_inode()). Unlike -before, if you are using inode-associated metadata buffers (i.e. -mark_buffer_dirty_inode()), it's your responsibility to call -invalidate_inode_buffers() before clear_inode(). - - NOTE: checking i_nlink in the beginning of ->write_inode() and bailing out -if it's zero is not *and* *never* *had* *been* enough. Final unlink() and iput() -may happen while the inode is in the middle of ->write_inode(); e.g. if you blindly -free the on-disk inode, you may end up doing that while ->write_inode() is writing -to it. - ---- -[mandatory] - - .d_delete() now only advises the dcache as to whether or not to cache -unreferenced dentries, and is now only called when the dentry refcount goes to -0. Even on 0 refcount transition, it must be able to tolerate being called 0, -1, or more times (eg. constant, idempotent). - ---- -[mandatory] - - .d_compare() calling convention and locking rules are significantly -changed. Read updated documentation in Documentation/filesystems/vfs.txt (and -look at examples of other filesystems) for guidance. - ---- -[mandatory] - - .d_hash() calling convention and locking rules are significantly -changed. Read updated documentation in Documentation/filesystems/vfs.txt (and -look at examples of other filesystems) for guidance. - ---- -[mandatory] - dcache_lock is gone, replaced by fine grained locks. See fs/dcache.c -for details of what locks to replace dcache_lock with in order to protect -particular things. Most of the time, a filesystem only needs ->d_lock, which -protects *all* the dcache state of a given dentry. - --- -[mandatory] - - Filesystems must RCU-free their inodes, if they can have been accessed -via rcu-walk path walk (basically, if the file can have had a path name in the -vfs namespace). - - Even though i_dentry and i_rcu share storage in a union, we will -initialize the former in inode_init_always(), so just leave it alone in -the callback. It used to be necessary to clean it there, but not anymore -(starting at 3.2). - --- -[recommended] - vfs now tries to do path walking in "rcu-walk mode", which avoids -atomic operations and scalability hazards on dentries and inodes (see -Documentation/filesystems/path-lookup.txt). d_hash and d_compare changes -(above) are examples of the changes required to support this. For more complex -filesystem callbacks, the vfs drops out of rcu-walk mode before the fs call, so -no changes are required to the filesystem. However, this is costly and loses -the benefits of rcu-walk mode. We will begin to add filesystem callbacks that -are rcu-walk aware, shown below. Filesystems should take advantage of this -where possible. - --- -[mandatory] - d_revalidate is a callback that is made on every path element (if -the filesystem provides it), which requires dropping out of rcu-walk mode. This -may now be called in rcu-walk mode (nd->flags & LOOKUP_RCU). -ECHILD should be -returned if the filesystem cannot handle rcu-walk. See -Documentation/filesystems/vfs.txt for more details. - - permission is an inode permission check that is called on many or all -directory inodes on the way down a path walk (to check for exec permission). It -must now be rcu-walk aware (mask & MAY_NOT_BLOCK). See -Documentation/filesystems/vfs.txt for more details. - --- -[mandatory] - In ->fallocate() you must check the mode option passed in. If your -filesystem does not support hole punching (deallocating space in the middle of a -file) you must return -EOPNOTSUPP if FALLOC_FL_PUNCH_HOLE is set in mode. -Currently you can only have FALLOC_FL_PUNCH_HOLE with FALLOC_FL_KEEP_SIZE set, -so the i_size should not change when hole punching, even when puching the end of -a file off. - --- -[mandatory] - ->get_sb() is gone. Switch to use of ->mount(). Typically it's just -a matter of switching from calling get_sb_... to mount_... and changing the -function type. If you were doing it manually, just switch from setting ->mnt_root -to some pointer to returning that pointer. On errors return ERR_PTR(...). - --- -[mandatory] - ->permission() and generic_permission()have lost flags -argument; instead of passing IPERM_FLAG_RCU we add MAY_NOT_BLOCK into mask. - generic_permission() has also lost the check_acl argument; ACL checking -has been taken to VFS and filesystems need to provide a non-NULL ->i_op->get_acl -to read an ACL from disk. - --- -[mandatory] - If you implement your own ->llseek() you must handle SEEK_HOLE and -SEEK_DATA. You can hanle this by returning -EINVAL, but it would be nicer to -support it in some way. The generic handler assumes that the entire file is -data and there is a virtual hole at the end of the file. So if the provided -offset is less than i_size and SEEK_DATA is specified, return the same offset. -If the above is true for the offset and you are given SEEK_HOLE, return the end -of the file. If the offset is i_size or greater return -ENXIO in either case. - -[mandatory] - If you have your own ->fsync() you must make sure to call -filemap_write_and_wait_range() so that all dirty pages are synced out properly. -You must also keep in mind that ->fsync() is not called with i_mutex held -anymore, so if you require i_mutex locking you must make sure to take it and -release it yourself. - --- -[mandatory] - d_alloc_root() is gone, along with a lot of bugs caused by code -misusing it. Replacement: d_make_root(inode). The difference is, -d_make_root() drops the reference to inode if dentry allocation fails. - --- -[mandatory] - The witch is dead! Well, 2/3 of it, anyway. ->d_revalidate() and -->lookup() do *not* take struct nameidata anymore; just the flags. --- -[mandatory] - ->create() doesn't take struct nameidata *; unlike the previous -two, it gets "is it an O_EXCL or equivalent?" boolean argument. Note that -local filesystems can ignore tha argument - they are guaranteed that the -object doesn't exist. It's remote/distributed ones that might care... --- -[mandatory] - FS_REVAL_DOT is gone; if you used to have it, add ->d_weak_revalidate() -in your dentry operations instead. --- -[mandatory] - vfs_readdir() is gone; switch to iterate_dir() instead --- -[mandatory] - ->readdir() is gone now; switch to ->iterate() -[mandatory] - vfs_follow_link has been removed. Filesystems must use nd_set_link - from ->follow_link for normal symlinks, or nd_jump_link for magic - /proc/<pid> style links. --- -[mandatory] - iget5_locked()/ilookup5()/ilookup5_nowait() test() callback used to be - called with both ->i_lock and inode_hash_lock held; the former is *not* - taken anymore, so verify that your callbacks do not rely on it (none - of the in-tree instances did). inode_hash_lock is still held, - of course, so they are still serialized wrt removal from inode hash, - as well as wrt set() callback of iget5_locked(). --- -[mandatory] - d_materialise_unique() is gone; d_splice_alias() does everything you - need now. Remember that they have opposite orders of arguments ;-/ --- -[mandatory] - f_dentry is gone; use f_path.dentry, or, better yet, see if you can avoid - it entirely. --- -[mandatory] - never call ->read() and ->write() directly; use __vfs_{read,write} or - wrappers; instead of checking for ->write or ->read being NULL, look for - FMODE_CAN_{WRITE,READ} in file->f_mode. --- -[mandatory] - do _not_ use new_sync_{read,write} for ->read/->write; leave it NULL - instead. --- -[mandatory] - ->aio_read/->aio_write are gone. Use ->read_iter/->write_iter. ---- -[recommended] - for embedded ("fast") symlinks just set inode->i_link to wherever the - symlink body is and use simple_follow_link() as ->follow_link(). --- -[mandatory] - calling conventions for ->follow_link() have changed. Instead of returning - cookie and using nd_set_link() to store the body to traverse, we return - the body to traverse and store the cookie using explicit void ** argument. - nameidata isn't passed at all - nd_jump_link() doesn't need it and - nd_[gs]et_link() is gone. --- -[mandatory] - calling conventions for ->put_link() have changed. It gets inode instead of - dentry, it does not get nameidata at all and it gets called only when cookie - is non-NULL. Note that link body isn't available anymore, so if you need it, - store it as cookie. --- -[mandatory] - any symlink that might use page_follow_link_light/page_put_link() must - have inode_nohighmem(inode) called before anything might start playing with - its pagecache. No highmem pages should end up in the pagecache of such - symlinks. That includes any preseeding that might be done during symlink - creation. __page_symlink() will honour the mapping gfp flags, so once - you've done inode_nohighmem() it's safe to use, but if you allocate and - insert the page manually, make sure to use the right gfp flags. --- -[mandatory] - ->follow_link() is replaced with ->get_link(); same API, except that - * ->get_link() gets inode as a separate argument - * ->get_link() may be called in RCU mode - in that case NULL - dentry is passed --- -[mandatory] - ->get_link() gets struct delayed_call *done now, and should do - set_delayed_call() where it used to set *cookie. - ->put_link() is gone - just give the destructor to set_delayed_call() - in ->get_link(). --- -[mandatory] - ->getxattr() and xattr_handler.get() get dentry and inode passed separately. - dentry might be yet to be attached to inode, so do _not_ use its ->d_inode - in the instances. Rationale: !@#!@# security_d_instantiate() needs to be - called before we attach dentry to inode. --- -[mandatory] - symlinks are no longer the only inodes that do *not* have i_bdev/i_cdev/ - i_pipe/i_link union zeroed out at inode eviction. As the result, you can't - assume that non-NULL value in ->i_nlink at ->destroy_inode() implies that - it's a symlink. Checking ->i_mode is really needed now. In-tree we had - to fix shmem_destroy_callback() that used to take that kind of shortcut; - watch out, since that shortcut is no longer valid. --- -[mandatory] - ->i_mutex is replaced with ->i_rwsem now. inode_lock() et.al. work as - they used to - they just take it exclusive. However, ->lookup() may be - called with parent locked shared. Its instances must not - * use d_instantiate) and d_rehash() separately - use d_add() or - d_splice_alias() instead. - * use d_rehash() alone - call d_add(new_dentry, NULL) instead. - * in the unlikely case when (read-only) access to filesystem - data structures needs exclusion for some reason, arrange it - yourself. None of the in-tree filesystems needed that. - * rely on ->d_parent and ->d_name not changing after dentry has - been fed to d_add() or d_splice_alias(). Again, none of the - in-tree instances relied upon that. - We are guaranteed that lookups of the same name in the same directory - will not happen in parallel ("same" in the sense of your ->d_compare()). - Lookups on different names in the same directory can and do happen in - parallel now. --- -[recommended] - ->iterate_shared() is added; it's a parallel variant of ->iterate(). - Exclusion on struct file level is still provided (as well as that - between it and lseek on the same struct file), but if your directory - has been opened several times, you can get these called in parallel. - Exclusion between that method and all directory-modifying ones is - still provided, of course. - - Often enough ->iterate() can serve as ->iterate_shared() without any - changes - it is a read-only operation, after all. If you have any - per-inode or per-dentry in-core data structures modified by ->iterate(), - you might need something to serialize the access to them. If you - do dcache pre-seeding, you'll need to switch to d_alloc_parallel() for - that; look for in-tree examples. - - Old method is only used if the new one is absent; eventually it will - be removed. Switch while you still can; the old one won't stay. --- -[mandatory] - ->atomic_open() calls without O_CREAT may happen in parallel. --- -[mandatory] - ->setxattr() and xattr_handler.set() get dentry and inode passed separately. - dentry might be yet to be attached to inode, so do _not_ use its ->d_inode - in the instances. Rationale: !@#!@# security_d_instantiate() needs to be - called before we attach dentry to inode and !@#!@##!@$!$#!@#$!@$!@$ smack - ->d_instantiate() uses not just ->getxattr() but ->setxattr() as well. --- -[mandatory] - ->d_compare() doesn't get parent as a separate argument anymore. If you - used it for finding the struct super_block involved, dentry->d_sb will - work just as well; if it's something more complicated, use dentry->d_parent. - Just be careful not to assume that fetching it more than once will yield - the same value - in RCU mode it could change under you. --- -[mandatory] - ->rename() has an added flags argument. Any flags not handled by the - filesystem should result in EINVAL being returned. --- -[recommended] - ->readlink is optional for symlinks. Don't set, unless filesystem needs - to fake something for readlink(2). --- -[mandatory] - ->getattr() is now passed a struct path rather than a vfsmount and - dentry separately, and it now has request_mask and query_flags arguments - to specify the fields and sync type requested by statx. Filesystems not - supporting any statx-specific features may ignore the new arguments. --- -[mandatory] - ->atomic_open() calling conventions have changed. Gone is int *opened, - along with FILE_OPENED/FILE_CREATED. In place of those we have - FMODE_OPENED/FMODE_CREATED, set in file->f_mode. Additionally, return - value for 'called finish_no_open(), open it yourself' case has become - 0, not 1. Since finish_no_open() itself is returning 0 now, that part - does not need any changes in ->atomic_open() instances. --- -[mandatory] - alloc_file() has become static now; two wrappers are to be used instead. - alloc_file_pseudo(inode, vfsmount, name, flags, ops) is for the cases - when dentry needs to be created; that's the majority of old alloc_file() - users. Calling conventions: on success a reference to new struct file - is returned and callers reference to inode is subsumed by that. On - failure, ERR_PTR() is returned and no caller's references are affected, - so the caller needs to drop the inode reference it held. - alloc_file_clone(file, flags, ops) does not affect any caller's references. - On success you get a new struct file sharing the mount/dentry with the - original, on failure - ERR_PTR(). --- -[mandatory] - ->clone_file_range() and ->dedupe_file_range have been replaced with - ->remap_file_range(). See Documentation/filesystems/vfs.txt for more - information. --- -[recommended] - ->lookup() instances doing an equivalent of - if (IS_ERR(inode)) - return ERR_CAST(inode); - return d_splice_alias(inode, dentry); - don't need to bother with the check - d_splice_alias() will do the - right thing when given ERR_PTR(...) as inode. Moreover, passing NULL - inode to d_splice_alias() will also do the right thing (equivalent of - d_add(dentry, NULL); return NULL;), so that kind of special cases - also doesn't need a separate treatment. --- -[strongly recommended] - take the RCU-delayed parts of ->destroy_inode() into a new method - - ->free_inode(). If ->destroy_inode() becomes empty - all the better, - just get rid of it. Synchronous work (e.g. the stuff that can't - be done from an RCU callback, or any WARN_ON() where we want the - stack trace) *might* be movable to ->evict_inode(); however, - that goes only for the things that are not needed to balance something - done by ->alloc_inode(). IOW, if it's cleaning up the stuff that - might have accumulated over the life of in-core inode, ->evict_inode() - might be a fit. - - Rules for inode destruction: - * if ->destroy_inode() is non-NULL, it gets called - * if ->free_inode() is non-NULL, it gets scheduled by call_rcu() - * combination of NULL ->destroy_inode and NULL ->free_inode is - treated as NULL/free_inode_nonrcu, to preserve the compatibility. - - Note that the callback (be it via ->free_inode() or explicit call_rcu() - in ->destroy_inode()) is *NOT* ordered wrt superblock destruction; - as the matter of fact, the superblock and all associated structures - might be already gone. The filesystem driver is guaranteed to be still - there, but that's it. Freeing memory in the callback is fine; doing - more than that is possible, but requires a lot of care and is best - avoided. --- -[mandatory] - DCACHE_RCUACCESS is gone; having an RCU delay on dentry freeing is the - default. DCACHE_NORCU opts out, and only d_alloc_pseudo() has any - business doing so. --- -[mandatory] - d_alloc_pseudo() is internal-only; uses outside of alloc_file_pseudo() are - very suspect (and won't work in modules). Such uses are very likely to - be misspelled d_alloc_anon(). diff --git a/Documentation/filesystems/porting.rst b/Documentation/filesystems/porting.rst new file mode 100644 index 000000000000..f18506083ced --- /dev/null +++ b/Documentation/filesystems/porting.rst @@ -0,0 +1,852 @@ +==================== +Changes since 2.5.0: +==================== + +--- + +**recommended** + +New helpers: sb_bread(), sb_getblk(), sb_find_get_block(), set_bh(), +sb_set_blocksize() and sb_min_blocksize(). + +Use them. + +(sb_find_get_block() replaces 2.4's get_hash_table()) + +--- + +**recommended** + +New methods: ->alloc_inode() and ->destroy_inode(). + +Remove inode->u.foo_inode_i + +Declare:: + + struct foo_inode_info { + /* fs-private stuff */ + struct inode vfs_inode; + }; + static inline struct foo_inode_info *FOO_I(struct inode *inode) + { + return list_entry(inode, struct foo_inode_info, vfs_inode); + } + +Use FOO_I(inode) instead of &inode->u.foo_inode_i; + +Add foo_alloc_inode() and foo_destroy_inode() - the former should allocate +foo_inode_info and return the address of ->vfs_inode, the latter should free +FOO_I(inode) (see in-tree filesystems for examples). + +Make them ->alloc_inode and ->destroy_inode in your super_operations. + +Keep in mind that now you need explicit initialization of private data +typically between calling iget_locked() and unlocking the inode. + +At some point that will become mandatory. + +--- + +**mandatory** + +Change of file_system_type method (->read_super to ->get_sb) + +->read_super() is no more. Ditto for DECLARE_FSTYPE and DECLARE_FSTYPE_DEV. + +Turn your foo_read_super() into a function that would return 0 in case of +success and negative number in case of error (-EINVAL unless you have more +informative error value to report). Call it foo_fill_super(). Now declare:: + + int foo_get_sb(struct file_system_type *fs_type, + int flags, const char *dev_name, void *data, struct vfsmount *mnt) + { + return get_sb_bdev(fs_type, flags, dev_name, data, foo_fill_super, + mnt); + } + +(or similar with s/bdev/nodev/ or s/bdev/single/, depending on the kind of +filesystem). + +Replace DECLARE_FSTYPE... with explicit initializer and have ->get_sb set as +foo_get_sb. + +--- + +**mandatory** + +Locking change: ->s_vfs_rename_sem is taken only by cross-directory renames. +Most likely there is no need to change anything, but if you relied on +global exclusion between renames for some internal purpose - you need to +change your internal locking. Otherwise exclusion warranties remain the +same (i.e. parents and victim are locked, etc.). + +--- + +**informational** + +Now we have the exclusion between ->lookup() and directory removal (by +->rmdir() and ->rename()). If you used to need that exclusion and do +it by internal locking (most of filesystems couldn't care less) - you +can relax your locking. + +--- + +**mandatory** + +->lookup(), ->truncate(), ->create(), ->unlink(), ->mknod(), ->mkdir(), +->rmdir(), ->link(), ->lseek(), ->symlink(), ->rename() +and ->readdir() are called without BKL now. Grab it on entry, drop upon return +- that will guarantee the same locking you used to have. If your method or its +parts do not need BKL - better yet, now you can shift lock_kernel() and +unlock_kernel() so that they would protect exactly what needs to be +protected. + +--- + +**mandatory** + +BKL is also moved from around sb operations. BKL should have been shifted into +individual fs sb_op functions. If you don't need it, remove it. + +--- + +**informational** + +check for ->link() target not being a directory is done by callers. Feel +free to drop it... + +--- + +**informational** + +->link() callers hold ->i_mutex on the object we are linking to. Some of your +problems might be over... + +--- + +**mandatory** + +new file_system_type method - kill_sb(superblock). If you are converting +an existing filesystem, set it according to ->fs_flags:: + + FS_REQUIRES_DEV - kill_block_super + FS_LITTER - kill_litter_super + neither - kill_anon_super + +FS_LITTER is gone - just remove it from fs_flags. + +--- + +**mandatory** + +FS_SINGLE is gone (actually, that had happened back when ->get_sb() +went in - and hadn't been documented ;-/). Just remove it from fs_flags +(and see ->get_sb() entry for other actions). + +--- + +**mandatory** + +->setattr() is called without BKL now. Caller _always_ holds ->i_mutex, so +watch for ->i_mutex-grabbing code that might be used by your ->setattr(). +Callers of notify_change() need ->i_mutex now. + +--- + +**recommended** + +New super_block field ``struct export_operations *s_export_op`` for +explicit support for exporting, e.g. via NFS. The structure is fully +documented at its declaration in include/linux/fs.h, and in +Documentation/filesystems/nfs/exporting.rst. + +Briefly it allows for the definition of decode_fh and encode_fh operations +to encode and decode filehandles, and allows the filesystem to use +a standard helper function for decode_fh, and provide file-system specific +support for this helper, particularly get_parent. + +It is planned that this will be required for exporting once the code +settles down a bit. + +**mandatory** + +s_export_op is now required for exporting a filesystem. +isofs, ext2, ext3, resierfs, fat +can be used as examples of very different filesystems. + +--- + +**mandatory** + +iget4() and the read_inode2 callback have been superseded by iget5_locked() +which has the following prototype:: + + struct inode *iget5_locked(struct super_block *sb, unsigned long ino, + int (*test)(struct inode *, void *), + int (*set)(struct inode *, void *), + void *data); + +'test' is an additional function that can be used when the inode +number is not sufficient to identify the actual file object. 'set' +should be a non-blocking function that initializes those parts of a +newly created inode to allow the test function to succeed. 'data' is +passed as an opaque value to both test and set functions. + +When the inode has been created by iget5_locked(), it will be returned with the +I_NEW flag set and will still be locked. The filesystem then needs to finalize +the initialization. Once the inode is initialized it must be unlocked by +calling unlock_new_inode(). + +The filesystem is responsible for setting (and possibly testing) i_ino +when appropriate. There is also a simpler iget_locked function that +just takes the superblock and inode number as arguments and does the +test and set for you. + +e.g.:: + + inode = iget_locked(sb, ino); + if (inode->i_state & I_NEW) { + err = read_inode_from_disk(inode); + if (err < 0) { + iget_failed(inode); + return err; + } + unlock_new_inode(inode); + } + +Note that if the process of setting up a new inode fails, then iget_failed() +should be called on the inode to render it dead, and an appropriate error +should be passed back to the caller. + +--- + +**recommended** + +->getattr() finally getting used. See instances in nfs, minix, etc. + +--- + +**mandatory** + +->revalidate() is gone. If your filesystem had it - provide ->getattr() +and let it call whatever you had as ->revlidate() + (for symlinks that +had ->revalidate()) add calls in ->follow_link()/->readlink(). + +--- + +**mandatory** + +->d_parent changes are not protected by BKL anymore. Read access is safe +if at least one of the following is true: + + * filesystem has no cross-directory rename() + * we know that parent had been locked (e.g. we are looking at + ->d_parent of ->lookup() argument). + * we are called from ->rename(). + * the child's ->d_lock is held + +Audit your code and add locking if needed. Notice that any place that is +not protected by the conditions above is risky even in the old tree - you +had been relying on BKL and that's prone to screwups. Old tree had quite +a few holes of that kind - unprotected access to ->d_parent leading to +anything from oops to silent memory corruption. + +--- + +**mandatory** + +FS_NOMOUNT is gone. If you use it - just set SB_NOUSER in flags +(see rootfs for one kind of solution and bdev/socket/pipe for another). + +--- + +**recommended** + +Use bdev_read_only(bdev) instead of is_read_only(kdev). The latter +is still alive, but only because of the mess in drivers/s390/block/dasd.c. +As soon as it gets fixed is_read_only() will die. + +--- + +**mandatory** + +->permission() is called without BKL now. Grab it on entry, drop upon +return - that will guarantee the same locking you used to have. If +your method or its parts do not need BKL - better yet, now you can +shift lock_kernel() and unlock_kernel() so that they would protect +exactly what needs to be protected. + +--- + +**mandatory** + +->statfs() is now called without BKL held. BKL should have been +shifted into individual fs sb_op functions where it's not clear that +it's safe to remove it. If you don't need it, remove it. + +--- + +**mandatory** + +is_read_only() is gone; use bdev_read_only() instead. + +--- + +**mandatory** + +destroy_buffers() is gone; use invalidate_bdev(). + +--- + +**mandatory** + +fsync_dev() is gone; use fsync_bdev(). NOTE: lvm breakage is +deliberate; as soon as struct block_device * is propagated in a reasonable +way by that code fixing will become trivial; until then nothing can be +done. + +**mandatory** + +block truncatation on error exit from ->write_begin, and ->direct_IO +moved from generic methods (block_write_begin, cont_write_begin, +nobh_write_begin, blockdev_direct_IO*) to callers. Take a look at +ext2_write_failed and callers for an example. + +**mandatory** + +->truncate is gone. The whole truncate sequence needs to be +implemented in ->setattr, which is now mandatory for filesystems +implementing on-disk size changes. Start with a copy of the old inode_setattr +and vmtruncate, and the reorder the vmtruncate + foofs_vmtruncate sequence to +be in order of zeroing blocks using block_truncate_page or similar helpers, +size update and on finally on-disk truncation which should not fail. +setattr_prepare (which used to be inode_change_ok) now includes the size checks +for ATTR_SIZE and must be called in the beginning of ->setattr unconditionally. + +**mandatory** + +->clear_inode() and ->delete_inode() are gone; ->evict_inode() should +be used instead. It gets called whenever the inode is evicted, whether it has +remaining links or not. Caller does *not* evict the pagecache or inode-associated +metadata buffers; the method has to use truncate_inode_pages_final() to get rid +of those. Caller makes sure async writeback cannot be running for the inode while +(or after) ->evict_inode() is called. + +->drop_inode() returns int now; it's called on final iput() with +inode->i_lock held and it returns true if filesystems wants the inode to be +dropped. As before, generic_drop_inode() is still the default and it's been +updated appropriately. generic_delete_inode() is also alive and it consists +simply of return 1. Note that all actual eviction work is done by caller after +->drop_inode() returns. + +As before, clear_inode() must be called exactly once on each call of +->evict_inode() (as it used to be for each call of ->delete_inode()). Unlike +before, if you are using inode-associated metadata buffers (i.e. +mark_buffer_dirty_inode()), it's your responsibility to call +invalidate_inode_buffers() before clear_inode(). + +NOTE: checking i_nlink in the beginning of ->write_inode() and bailing out +if it's zero is not *and* *never* *had* *been* enough. Final unlink() and iput() +may happen while the inode is in the middle of ->write_inode(); e.g. if you blindly +free the on-disk inode, you may end up doing that while ->write_inode() is writing +to it. + +--- + +**mandatory** + +.d_delete() now only advises the dcache as to whether or not to cache +unreferenced dentries, and is now only called when the dentry refcount goes to +0. Even on 0 refcount transition, it must be able to tolerate being called 0, +1, or more times (eg. constant, idempotent). + +--- + +**mandatory** + +.d_compare() calling convention and locking rules are significantly +changed. Read updated documentation in Documentation/filesystems/vfs.rst (and +look at examples of other filesystems) for guidance. + +--- + +**mandatory** + +.d_hash() calling convention and locking rules are significantly +changed. Read updated documentation in Documentation/filesystems/vfs.rst (and +look at examples of other filesystems) for guidance. + +--- + +**mandatory** + +dcache_lock is gone, replaced by fine grained locks. See fs/dcache.c +for details of what locks to replace dcache_lock with in order to protect +particular things. Most of the time, a filesystem only needs ->d_lock, which +protects *all* the dcache state of a given dentry. + +--- + +**mandatory** + +Filesystems must RCU-free their inodes, if they can have been accessed +via rcu-walk path walk (basically, if the file can have had a path name in the +vfs namespace). + +Even though i_dentry and i_rcu share storage in a union, we will +initialize the former in inode_init_always(), so just leave it alone in +the callback. It used to be necessary to clean it there, but not anymore +(starting at 3.2). + +--- + +**recommended** + +vfs now tries to do path walking in "rcu-walk mode", which avoids +atomic operations and scalability hazards on dentries and inodes (see +Documentation/filesystems/path-lookup.txt). d_hash and d_compare changes +(above) are examples of the changes required to support this. For more complex +filesystem callbacks, the vfs drops out of rcu-walk mode before the fs call, so +no changes are required to the filesystem. However, this is costly and loses +the benefits of rcu-walk mode. We will begin to add filesystem callbacks that +are rcu-walk aware, shown below. Filesystems should take advantage of this +where possible. + +--- + +**mandatory** + +d_revalidate is a callback that is made on every path element (if +the filesystem provides it), which requires dropping out of rcu-walk mode. This +may now be called in rcu-walk mode (nd->flags & LOOKUP_RCU). -ECHILD should be +returned if the filesystem cannot handle rcu-walk. See +Documentation/filesystems/vfs.rst for more details. + +permission is an inode permission check that is called on many or all +directory inodes on the way down a path walk (to check for exec permission). It +must now be rcu-walk aware (mask & MAY_NOT_BLOCK). See +Documentation/filesystems/vfs.rst for more details. + +--- + +**mandatory** + +In ->fallocate() you must check the mode option passed in. If your +filesystem does not support hole punching (deallocating space in the middle of a +file) you must return -EOPNOTSUPP if FALLOC_FL_PUNCH_HOLE is set in mode. +Currently you can only have FALLOC_FL_PUNCH_HOLE with FALLOC_FL_KEEP_SIZE set, +so the i_size should not change when hole punching, even when puching the end of +a file off. + +--- + +**mandatory** + +->get_sb() is gone. Switch to use of ->mount(). Typically it's just +a matter of switching from calling ``get_sb_``... to ``mount_``... and changing +the function type. If you were doing it manually, just switch from setting +->mnt_root to some pointer to returning that pointer. On errors return +ERR_PTR(...). + +--- + +**mandatory** + +->permission() and generic_permission()have lost flags +argument; instead of passing IPERM_FLAG_RCU we add MAY_NOT_BLOCK into mask. + +generic_permission() has also lost the check_acl argument; ACL checking +has been taken to VFS and filesystems need to provide a non-NULL ->i_op->get_acl +to read an ACL from disk. + +--- + +**mandatory** + +If you implement your own ->llseek() you must handle SEEK_HOLE and +SEEK_DATA. You can hanle this by returning -EINVAL, but it would be nicer to +support it in some way. The generic handler assumes that the entire file is +data and there is a virtual hole at the end of the file. So if the provided +offset is less than i_size and SEEK_DATA is specified, return the same offset. +If the above is true for the offset and you are given SEEK_HOLE, return the end +of the file. If the offset is i_size or greater return -ENXIO in either case. + +**mandatory** + +If you have your own ->fsync() you must make sure to call +filemap_write_and_wait_range() so that all dirty pages are synced out properly. +You must also keep in mind that ->fsync() is not called with i_mutex held +anymore, so if you require i_mutex locking you must make sure to take it and +release it yourself. + +--- + +**mandatory** + +d_alloc_root() is gone, along with a lot of bugs caused by code +misusing it. Replacement: d_make_root(inode). On success d_make_root(inode) +allocates and returns a new dentry instantiated with the passed in inode. +On failure NULL is returned and the passed in inode is dropped so the reference +to inode is consumed in all cases and failure handling need not do any cleanup +for the inode. If d_make_root(inode) is passed a NULL inode it returns NULL +and also requires no further error handling. Typical usage is:: + + inode = foofs_new_inode(....); + s->s_root = d_make_root(inode); + if (!s->s_root) + /* Nothing needed for the inode cleanup */ + return -ENOMEM; + ... + +--- + +**mandatory** + +The witch is dead! Well, 2/3 of it, anyway. ->d_revalidate() and +->lookup() do *not* take struct nameidata anymore; just the flags. + +--- + +**mandatory** + +->create() doesn't take ``struct nameidata *``; unlike the previous +two, it gets "is it an O_EXCL or equivalent?" boolean argument. Note that +local filesystems can ignore tha argument - they are guaranteed that the +object doesn't exist. It's remote/distributed ones that might care... + +--- + +**mandatory** + +FS_REVAL_DOT is gone; if you used to have it, add ->d_weak_revalidate() +in your dentry operations instead. + +--- + +**mandatory** + +vfs_readdir() is gone; switch to iterate_dir() instead + +--- + +**mandatory** + +->readdir() is gone now; switch to ->iterate() + +**mandatory** + +vfs_follow_link has been removed. Filesystems must use nd_set_link +from ->follow_link for normal symlinks, or nd_jump_link for magic +/proc/<pid> style links. + +--- + +**mandatory** + +iget5_locked()/ilookup5()/ilookup5_nowait() test() callback used to be +called with both ->i_lock and inode_hash_lock held; the former is *not* +taken anymore, so verify that your callbacks do not rely on it (none +of the in-tree instances did). inode_hash_lock is still held, +of course, so they are still serialized wrt removal from inode hash, +as well as wrt set() callback of iget5_locked(). + +--- + +**mandatory** + +d_materialise_unique() is gone; d_splice_alias() does everything you +need now. Remember that they have opposite orders of arguments ;-/ + +--- + +**mandatory** + +f_dentry is gone; use f_path.dentry, or, better yet, see if you can avoid +it entirely. + +--- + +**mandatory** + +never call ->read() and ->write() directly; use __vfs_{read,write} or +wrappers; instead of checking for ->write or ->read being NULL, look for +FMODE_CAN_{WRITE,READ} in file->f_mode. + +--- + +**mandatory** + +do _not_ use new_sync_{read,write} for ->read/->write; leave it NULL +instead. + +--- + +**mandatory** + ->aio_read/->aio_write are gone. Use ->read_iter/->write_iter. + +--- + +**recommended** + +for embedded ("fast") symlinks just set inode->i_link to wherever the +symlink body is and use simple_follow_link() as ->follow_link(). + +--- + +**mandatory** + +calling conventions for ->follow_link() have changed. Instead of returning +cookie and using nd_set_link() to store the body to traverse, we return +the body to traverse and store the cookie using explicit void ** argument. +nameidata isn't passed at all - nd_jump_link() doesn't need it and +nd_[gs]et_link() is gone. + +--- + +**mandatory** + +calling conventions for ->put_link() have changed. It gets inode instead of +dentry, it does not get nameidata at all and it gets called only when cookie +is non-NULL. Note that link body isn't available anymore, so if you need it, +store it as cookie. + +--- + +**mandatory** + +any symlink that might use page_follow_link_light/page_put_link() must +have inode_nohighmem(inode) called before anything might start playing with +its pagecache. No highmem pages should end up in the pagecache of such +symlinks. That includes any preseeding that might be done during symlink +creation. __page_symlink() will honour the mapping gfp flags, so once +you've done inode_nohighmem() it's safe to use, but if you allocate and +insert the page manually, make sure to use the right gfp flags. + +--- + +**mandatory** + +->follow_link() is replaced with ->get_link(); same API, except that + + * ->get_link() gets inode as a separate argument + * ->get_link() may be called in RCU mode - in that case NULL + dentry is passed + +--- + +**mandatory** + +->get_link() gets struct delayed_call ``*done`` now, and should do +set_delayed_call() where it used to set ``*cookie``. + +->put_link() is gone - just give the destructor to set_delayed_call() +in ->get_link(). + +--- + +**mandatory** + +->getxattr() and xattr_handler.get() get dentry and inode passed separately. +dentry might be yet to be attached to inode, so do _not_ use its ->d_inode +in the instances. Rationale: !@#!@# security_d_instantiate() needs to be +called before we attach dentry to inode. + +--- + +**mandatory** + +symlinks are no longer the only inodes that do *not* have i_bdev/i_cdev/ +i_pipe/i_link union zeroed out at inode eviction. As the result, you can't +assume that non-NULL value in ->i_nlink at ->destroy_inode() implies that +it's a symlink. Checking ->i_mode is really needed now. In-tree we had +to fix shmem_destroy_callback() that used to take that kind of shortcut; +watch out, since that shortcut is no longer valid. + +--- + +**mandatory** + +->i_mutex is replaced with ->i_rwsem now. inode_lock() et.al. work as +they used to - they just take it exclusive. However, ->lookup() may be +called with parent locked shared. Its instances must not + + * use d_instantiate) and d_rehash() separately - use d_add() or + d_splice_alias() instead. + * use d_rehash() alone - call d_add(new_dentry, NULL) instead. + * in the unlikely case when (read-only) access to filesystem + data structures needs exclusion for some reason, arrange it + yourself. None of the in-tree filesystems needed that. + * rely on ->d_parent and ->d_name not changing after dentry has + been fed to d_add() or d_splice_alias(). Again, none of the + in-tree instances relied upon that. + +We are guaranteed that lookups of the same name in the same directory +will not happen in parallel ("same" in the sense of your ->d_compare()). +Lookups on different names in the same directory can and do happen in +parallel now. + +--- + +**recommended** + +->iterate_shared() is added; it's a parallel variant of ->iterate(). +Exclusion on struct file level is still provided (as well as that +between it and lseek on the same struct file), but if your directory +has been opened several times, you can get these called in parallel. +Exclusion between that method and all directory-modifying ones is +still provided, of course. + +Often enough ->iterate() can serve as ->iterate_shared() without any +changes - it is a read-only operation, after all. If you have any +per-inode or per-dentry in-core data structures modified by ->iterate(), +you might need something to serialize the access to them. If you +do dcache pre-seeding, you'll need to switch to d_alloc_parallel() for +that; look for in-tree examples. + +Old method is only used if the new one is absent; eventually it will +be removed. Switch while you still can; the old one won't stay. + +--- + +**mandatory** + +->atomic_open() calls without O_CREAT may happen in parallel. + +--- + +**mandatory** + +->setxattr() and xattr_handler.set() get dentry and inode passed separately. +dentry might be yet to be attached to inode, so do _not_ use its ->d_inode +in the instances. Rationale: !@#!@# security_d_instantiate() needs to be +called before we attach dentry to inode and !@#!@##!@$!$#!@#$!@$!@$ smack +->d_instantiate() uses not just ->getxattr() but ->setxattr() as well. + +--- + +**mandatory** + +->d_compare() doesn't get parent as a separate argument anymore. If you +used it for finding the struct super_block involved, dentry->d_sb will +work just as well; if it's something more complicated, use dentry->d_parent. +Just be careful not to assume that fetching it more than once will yield +the same value - in RCU mode it could change under you. + +--- + +**mandatory** + +->rename() has an added flags argument. Any flags not handled by the +filesystem should result in EINVAL being returned. + +--- + + +**recommended** + +->readlink is optional for symlinks. Don't set, unless filesystem needs +to fake something for readlink(2). + +--- + +**mandatory** + +->getattr() is now passed a struct path rather than a vfsmount and +dentry separately, and it now has request_mask and query_flags arguments +to specify the fields and sync type requested by statx. Filesystems not +supporting any statx-specific features may ignore the new arguments. + +--- + +**mandatory** + +->atomic_open() calling conventions have changed. Gone is ``int *opened``, +along with FILE_OPENED/FILE_CREATED. In place of those we have +FMODE_OPENED/FMODE_CREATED, set in file->f_mode. Additionally, return +value for 'called finish_no_open(), open it yourself' case has become +0, not 1. Since finish_no_open() itself is returning 0 now, that part +does not need any changes in ->atomic_open() instances. + +--- + +**mandatory** + +alloc_file() has become static now; two wrappers are to be used instead. +alloc_file_pseudo(inode, vfsmount, name, flags, ops) is for the cases +when dentry needs to be created; that's the majority of old alloc_file() +users. Calling conventions: on success a reference to new struct file +is returned and callers reference to inode is subsumed by that. On +failure, ERR_PTR() is returned and no caller's references are affected, +so the caller needs to drop the inode reference it held. +alloc_file_clone(file, flags, ops) does not affect any caller's references. +On success you get a new struct file sharing the mount/dentry with the +original, on failure - ERR_PTR(). + +--- + +**mandatory** + +->clone_file_range() and ->dedupe_file_range have been replaced with +->remap_file_range(). See Documentation/filesystems/vfs.rst for more +information. + +--- + +**recommended** + +->lookup() instances doing an equivalent of:: + + if (IS_ERR(inode)) + return ERR_CAST(inode); + return d_splice_alias(inode, dentry); + +don't need to bother with the check - d_splice_alias() will do the +right thing when given ERR_PTR(...) as inode. Moreover, passing NULL +inode to d_splice_alias() will also do the right thing (equivalent of +d_add(dentry, NULL); return NULL;), so that kind of special cases +also doesn't need a separate treatment. + +--- + +**strongly recommended** + +take the RCU-delayed parts of ->destroy_inode() into a new method - +->free_inode(). If ->destroy_inode() becomes empty - all the better, +just get rid of it. Synchronous work (e.g. the stuff that can't +be done from an RCU callback, or any WARN_ON() where we want the +stack trace) *might* be movable to ->evict_inode(); however, +that goes only for the things that are not needed to balance something +done by ->alloc_inode(). IOW, if it's cleaning up the stuff that +might have accumulated over the life of in-core inode, ->evict_inode() +might be a fit. + +Rules for inode destruction: + + * if ->destroy_inode() is non-NULL, it gets called + * if ->free_inode() is non-NULL, it gets scheduled by call_rcu() + * combination of NULL ->destroy_inode and NULL ->free_inode is + treated as NULL/free_inode_nonrcu, to preserve the compatibility. + +Note that the callback (be it via ->free_inode() or explicit call_rcu() +in ->destroy_inode()) is *NOT* ordered wrt superblock destruction; +as the matter of fact, the superblock and all associated structures +might be already gone. The filesystem driver is guaranteed to be still +there, but that's it. Freeing memory in the callback is fine; doing +more than that is possible, but requires a lot of care and is best +avoided. + +--- + +**mandatory** + +DCACHE_RCUACCESS is gone; having an RCU delay on dentry freeing is the +default. DCACHE_NORCU opts out, and only d_alloc_pseudo() has any +business doing so. + +--- + +**mandatory** + +d_alloc_pseudo() is internal-only; uses outside of alloc_file_pseudo() are +very suspect (and won't work in modules). Such uses are very likely to +be misspelled d_alloc_anon(). diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt index 66cad5c86171..99ca040e3f90 100644 --- a/Documentation/filesystems/proc.txt +++ b/Documentation/filesystems/proc.txt @@ -45,6 +45,7 @@ Table of Contents 3.9 /proc/<pid>/map_files - Information about memory mapped files 3.10 /proc/<pid>/timerslack_ns - Task timerslack value 3.11 /proc/<pid>/patch_state - Livepatch patch operation state + 3.12 /proc/<pid>/arch_status - Task architecture specific information 4 Configuring procfs 4.1 Mount options @@ -153,9 +154,11 @@ Table 1-1: Process specific entries in /proc symbol the task is blocked in - or "0" if not blocked. pagemap Page table stack Report full stack trace, enable via CONFIG_STACKTRACE - smaps an extension based on maps, showing the memory consumption of + smaps An extension based on maps, showing the memory consumption of each mapping and flags associated with it - numa_maps an extension based on maps, showing the memory locality and + smaps_rollup Accumulated smaps stats for all mappings of the process. This + can be derived from smaps, but is faster and more convenient + numa_maps An extension based on maps, showing the memory locality and binding policy as well as mem usage (in pages) of each mapping. .............................................................................. @@ -365,7 +368,7 @@ Table 1-4: Contents of the stat files (as of 2.6.30-rc7) exit_code the thread's exit_code in the form reported by the waitpid system call .............................................................................. -The /proc/PID/maps file containing the currently mapped memory regions and +The /proc/PID/maps file contains the currently mapped memory regions and their access permissions. The format is: @@ -416,11 +419,14 @@ is not associated with a file: or if empty, the mapping is anonymous. The /proc/PID/smaps is an extension based on maps, showing the memory -consumption for each of the process's mappings. For each of mappings there -is a series of lines such as the following: +consumption for each of the process's mappings. For each mapping (aka Virtual +Memory Area, or VMA) there is a series of lines such as the following: 08048000-080bc000 r-xp 00000000 03:02 13130 /bin/bash + Size: 1084 kB +KernelPageSize: 4 kB +MMUPageSize: 4 kB Rss: 892 kB Pss: 374 kB Shared_Clean: 892 kB @@ -442,11 +448,14 @@ Locked: 0 kB THPeligible: 0 VmFlags: rd ex mr mw me dw -the first of these lines shows the same information as is displayed for the -mapping in /proc/PID/maps. The remaining lines show the size of the mapping -(size), the amount of the mapping that is currently resident in RAM (RSS), the -process' proportional share of this mapping (PSS), the number of clean and -dirty private pages in the mapping. +The first of these lines shows the same information as is displayed for the +mapping in /proc/PID/maps. Following lines show the size of the mapping +(size); the size of each page allocated when backing a VMA (KernelPageSize), +which is usually the same as the size in the page table entries; the page size +used by the MMU when backing a VMA (in most cases, the same as KernelPageSize); +the amount of the mapping that is currently resident in RAM (RSS); the +process' proportional share of this mapping (PSS); and the number of clean and +dirty shared and private pages in the mapping. The "proportional set size" (PSS) of a process is the count of pages it has in memory, where each page is divided by the number of processes sharing it. @@ -477,8 +486,8 @@ replaced by copy-on-write) part of the underlying shmem object out on swap. "SwapPss" shows proportional swap share of this mapping. Unlike "Swap", this does not take into account swapped out page of underlying shmem objects. "Locked" indicates whether the mapping is locked in memory or not. -"THPeligible" indicates whether the mapping is eligible for THP pages - 1 if -true, 0 otherwise. +"THPeligible" indicates whether the mapping is eligible for allocating THP +pages - 1 if true, 0 otherwise. It just shows the current status. "VmFlags" field deserves a separate description. This member represents the kernel flags associated with the particular virtual memory area in two letter encoded @@ -531,6 +540,19 @@ guarantees: 2) If there is something at a given vaddr during the entirety of the life of the smaps/maps walk, there will be some output for it. +The /proc/PID/smaps_rollup file includes the same fields as /proc/PID/smaps, +but their values are the sums of the corresponding values for all mappings of +the process. Additionally, it contains these fields: + +Pss_Anon +Pss_File +Pss_Shmem + +They represent the proportional shares of anonymous, file, and shmem pages, as +described for smaps above. These fields are omitted in smaps since each +mapping identifies the type (anon, file, or shmem) of all pages it contains. +Thus all information in smaps_rollup can be derived from smaps, but at a +significantly higher cost. The /proc/PID/clear_refs is used to reset the PG_Referenced and ACCESSED/YOUNG bits on both physical and virtual pages associated with a process, and the @@ -1478,7 +1500,7 @@ review the kernel documentation in the directory /usr/src/linux/Documentation. This chapter is heavily based on the documentation included in the pre 2.2 kernels, and became part of it in version 2.2.1 of the Linux kernel. -Please see: Documentation/sysctl/ directory for descriptions of these +Please see: Documentation/admin-guide/sysctl/ directory for descriptions of these entries. ------------------------------------------------------------------------------ @@ -1948,6 +1970,45 @@ patched. If the patch is being enabled, then the task has already been patched. If the patch is being disabled, then the task hasn't been unpatched yet. +3.12 /proc/<pid>/arch_status - task architecture specific status +------------------------------------------------------------------- +When CONFIG_PROC_PID_ARCH_STATUS is enabled, this file displays the +architecture specific status of the task. + +Example +------- + $ cat /proc/6753/arch_status + AVX512_elapsed_ms: 8 + +Description +----------- + +x86 specific entries: +--------------------- + AVX512_elapsed_ms: + ------------------ + If AVX512 is supported on the machine, this entry shows the milliseconds + elapsed since the last time AVX512 usage was recorded. The recording + happens on a best effort basis when a task is scheduled out. This means + that the value depends on two factors: + + 1) The time which the task spent on the CPU without being scheduled + out. With CPU isolation and a single runnable task this can take + several seconds. + + 2) The time since the task was scheduled out last. Depending on the + reason for being scheduled out (time slice exhausted, syscall ...) + this can be arbitrary long time. + + As a consequence the value cannot be considered precise and authoritative + information. The application which uses this information has to be aware + of the overall scenario on the system in order to determine whether a + task is a real AVX512 user or not. Precise information can be obtained + with performance counters. + + A special value of '-1' indicates that no AVX512 usage was recorded, thus + the task is unlikely an AVX512 user, but depends on the workload and the + scheduling scenario, it also could be a false negative mentioned above. ------------------------------------------------------------------------------ Configuring procfs diff --git a/Documentation/filesystems/ramfs-rootfs-initramfs.txt b/Documentation/filesystems/ramfs-rootfs-initramfs.txt index 79637d227e85..97d42ccaa92d 100644 --- a/Documentation/filesystems/ramfs-rootfs-initramfs.txt +++ b/Documentation/filesystems/ramfs-rootfs-initramfs.txt @@ -105,7 +105,7 @@ All this differs from the old initrd in several ways: - The old initrd file was a gzipped filesystem image (in some file format, such as ext2, that needed a driver built into the kernel), while the new initramfs archive is a gzipped cpio archive (like tar only simpler, - see cpio(1) and Documentation/early-userspace/buffer-format.txt). The + see cpio(1) and Documentation/driver-api/early-userspace/buffer-format.rst). The kernel's cpio extraction code is not only extremely small, it's also __init text and data that can be discarded during the boot process. @@ -159,7 +159,7 @@ One advantage of the configuration file is that root access is not required to set permissions or create device nodes in the new archive. (Note that those two example "file" entries expect to find files named "init.sh" and "busybox" in a directory called "initramfs", under the linux-2.6.* directory. See -Documentation/early-userspace/README for more details.) +Documentation/driver-api/early-userspace/early_userspace_support.rst for more details.) The kernel does not depend on external cpio tools. If you specify a directory instead of a configuration file, the kernel's build infrastructure diff --git a/Documentation/filesystems/sysfs.txt b/Documentation/filesystems/sysfs.txt index 5b5311f9358d..ddf15b1b0d5a 100644 --- a/Documentation/filesystems/sysfs.txt +++ b/Documentation/filesystems/sysfs.txt @@ -319,7 +319,7 @@ quick way to lookup the sysfs interface for a device from the result of a stat(2) operation. More information can driver-model specific features can be found in -Documentation/driver-model/. +Documentation/driver-api/driver-model/. TODO: Finish this section. diff --git a/Documentation/filesystems/tmpfs.txt b/Documentation/filesystems/tmpfs.txt index d06e9a59a9f4..5ecbc03e6b2f 100644 --- a/Documentation/filesystems/tmpfs.txt +++ b/Documentation/filesystems/tmpfs.txt @@ -98,7 +98,7 @@ A memory policy with a valid NodeList will be saved, as specified, for use at file creation time. When a task allocates a file in the file system, the mount option memory policy will be applied with a NodeList, if any, modified by the calling task's cpuset constraints -[See Documentation/cgroup-v1/cpusets.txt] and any optional flags, listed +[See Documentation/admin-guide/cgroup-v1/cpusets.rst] and any optional flags, listed below. If the resulting NodeLists is the empty set, the effective memory policy for the file will revert to "default" policy. diff --git a/Documentation/filesystems/ubifs-authentication.md b/Documentation/filesystems/ubifs-authentication.rst index 028b3e2e25f9..6a9584f6ff46 100644 --- a/Documentation/filesystems/ubifs-authentication.md +++ b/Documentation/filesystems/ubifs-authentication.rst @@ -1,8 +1,11 @@ -% UBIFS Authentication -% sigma star gmbh -% 2018 +:orphan: -# Introduction +.. UBIFS Authentication +.. sigma star gmbh +.. 2018 + +Introduction +============ UBIFS utilizes the fscrypt framework to provide confidentiality for file contents and file names. This prevents attacks where an attacker is able to @@ -33,7 +36,8 @@ existing features like key derivation can be utilized. It should however also be possible to use UBIFS authentication without using encryption. -## MTD, UBI & UBIFS +MTD, UBI & UBIFS +---------------- On Linux, the MTD (Memory Technology Devices) subsystem provides a uniform interface to access raw flash devices. One of the more prominent subsystems that @@ -47,7 +51,7 @@ UBIFS is a filesystem for raw flash which operates on top of UBI. Thus, wear leveling and some flash specifics are left to UBI, while UBIFS focuses on scalability, performance and recoverability. - +:: +------------+ +*******+ +-----------+ +-----+ | | * UBIFS * | UBI-BLOCK | | ... | @@ -84,7 +88,8 @@ persisted onto the flash directly. More details on UBIFS can also be found in [UBIFS-WP]. -### UBIFS Index & Tree Node Cache +UBIFS Index & Tree Node Cache +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Basic on-flash UBIFS entities are called *nodes*. UBIFS knows different types of nodes. Eg. data nodes (`struct ubifs_data_node`) which store chunks of file @@ -118,17 +123,18 @@ on-flash filesystem structures like the index. On every commit, the TNC nodes marked as dirty are written to the flash to update the persisted index. -### Journal +Journal +~~~~~~~ To avoid wearing out the flash, the index is only persisted (*commited*) when -certain conditions are met (eg. `fsync(2)`). The journal is used to record +certain conditions are met (eg. ``fsync(2)``). The journal is used to record any changes (in form of inode nodes, data nodes etc.) between commits of the index. During mount, the journal is read from the flash and replayed onto the TNC (which will be created on-demand from the on-flash index). UBIFS reserves a bunch of LEBs just for the journal called *log area*. The amount of log area LEBs is configured on filesystem creation (using -`mkfs.ubifs`) and stored in the superblock node. The log area contains only +``mkfs.ubifs``) and stored in the superblock node. The log area contains only two types of nodes: *reference nodes* and *commit start nodes*. A commit start node is written whenever an index commit is performed. Reference nodes are written on every journal update. Each reference node points to the position of @@ -152,6 +158,7 @@ done for the last referenced LEB of the journal. Only this can become corrupt because of a power cut. If the recovery fails, UBIFS will not mount. An error for every other LEB will directly cause UBIFS to fail the mount operation. +:: | ---- LOG AREA ---- | ---------- MAIN AREA ------------ | @@ -172,10 +179,11 @@ for every other LEB will directly cause UBIFS to fail the mount operation. containing their buds -### LEB Property Tree/Table +LEB Property Tree/Table +~~~~~~~~~~~~~~~~~~~~~~~ The LEB property tree is used to store per-LEB information. This includes the -LEB type and amount of free and *dirty* (old, obsolete content) space [1] on +LEB type and amount of free and *dirty* (old, obsolete content) space [1]_ on the LEB. The type is important, because UBIFS never mixes index nodes with data nodes on a single LEB and thus each LEB has a specific purpose. This again is useful for free space calculations. See [UBIFS-WP] for more details. @@ -185,19 +193,21 @@ index. Due to its smaller size it is always written as one chunk on every commit. Thus, saving the LPT is an atomic operation. -[1] Since LEBs can only be appended and never overwritten, there is a -difference between free space ie. the remaining space left on the LEB to be -written to without erasing it and previously written content that is obsolete -but can't be overwritten without erasing the full LEB. +.. [1] Since LEBs can only be appended and never overwritten, there is a + difference between free space ie. the remaining space left on the LEB to be + written to without erasing it and previously written content that is obsolete + but can't be overwritten without erasing the full LEB. -# UBIFS Authentication +UBIFS Authentication +==================== This chapter introduces UBIFS authentication which enables UBIFS to verify the authenticity and integrity of metadata and file contents stored on flash. -## Threat Model +Threat Model +------------ UBIFS authentication enables detection of offline data modification. While it does not prevent it, it enables (trusted) code to check the integrity and @@ -224,7 +234,8 @@ Additional measures like secure boot and trusted boot have to be taken to ensure that only trusted code is executed on a device. -## Authentication +Authentication +-------------- To be able to fully trust data read from flash, all UBIFS data structures stored on flash are authenticated. That is: @@ -236,7 +247,8 @@ stored on flash are authenticated. That is: - The LPT which stores UBI LEB metadata which UBIFS uses for free space accounting -### Index Authentication +Index Authentication +~~~~~~~~~~~~~~~~~~~~ Through UBIFS' concept of a wandering tree, it already takes care of only updating and persisting changed parts from leaf node up to the root node @@ -260,6 +272,7 @@ include a hash. All other types of nodes will remain unchanged. This reduces the storage overhead which is precious for users of UBIFS (ie. embedded devices). +:: +---------------+ | Master Node | @@ -303,7 +316,8 @@ hashes to index nodes does not change this since each hash will be persisted atomically together with its respective node. -### Journal Authentication +Journal Authentication +~~~~~~~~~~~~~~~~~~~~~~ The journal is authenticated too. Since the journal is continuously written it is necessary to also add authentication information frequently to the @@ -316,7 +330,7 @@ of the hash chain. That way a journal can be authenticated up to the last authentication node. The tail of the journal which may not have a authentication node cannot be authenticated and is skipped during journal replay. -We get this picture for journal authentication: +We get this picture for journal authentication:: ,,,,,,,, ,......,........................................... @@ -352,7 +366,8 @@ the superblock struct. The superblock node is stored in LEB 0 and is only modified on feature flag or similar changes, but never on file changes. -### LPT Authentication +LPT Authentication +~~~~~~~~~~~~~~~~~~ The location of the LPT root node on the flash is stored in the UBIFS master node. Since the LPT is written and read atomically on every commit, there is @@ -363,7 +378,8 @@ be verified by verifying the authenticity of the master node and comparing the LTP hash stored there with the hash computed from the read on-flash LPT. -## Key Management +Key Management +-------------- For simplicity, UBIFS authentication uses a single key to compute the HMACs of superblock, master, commit start and reference nodes. This key has to be @@ -399,7 +415,8 @@ approach is similar to the approach proposed for fscrypt encryption policy v2 [FSCRYPT-POLICY2]. -# Future Extensions +Future Extensions +================= In certain cases where a vendor wants to provide an authenticated filesystem image to customers, it should be possible to do so without sharing the secret @@ -411,15 +428,16 @@ to the way the IMA/EVM subsystem deals with such situations. The HMAC key will then have to be provided beforehand in the normal way. -# References +References +========== [CRYPTSETUP2] http://www.saout.de/pipermail/dm-crypt/2017-November/005745.html [DMC-CBC-ATTACK] http://www.jakoblell.com/blog/2013/12/22/practical-malleability-attack-against-cbc-encrypted-luks-partitions/ -[DM-INTEGRITY] https://www.kernel.org/doc/Documentation/device-mapper/dm-integrity.txt +[DM-INTEGRITY] https://www.kernel.org/doc/Documentation/device-mapper/dm-integrity.rst -[DM-VERITY] https://www.kernel.org/doc/Documentation/device-mapper/verity.txt +[DM-VERITY] https://www.kernel.org/doc/Documentation/device-mapper/verity.rst [FSCRYPT-POLICY2] https://www.spinics.net/lists/linux-ext4/msg58710.html diff --git a/Documentation/filesystems/ufs.txt b/Documentation/filesystems/ufs.txt deleted file mode 100644 index 7a602adeca2b..000000000000 --- a/Documentation/filesystems/ufs.txt +++ /dev/null @@ -1,60 +0,0 @@ -USING UFS -========= - -mount -t ufs -o ufstype=type_of_ufs device dir - - -UFS OPTIONS -=========== - -ufstype=type_of_ufs - UFS is a file system widely used in different operating systems. - The problem are differences among implementations. Features of - some implementations are undocumented, so its hard to recognize - type of ufs automatically. That's why user must specify type of - ufs manually by mount option ufstype. Possible values are: - - old old format of ufs - default value, supported as read-only - - 44bsd used in FreeBSD, NetBSD, OpenBSD - supported as read-write - - ufs2 used in FreeBSD 5.x - supported as read-write - - 5xbsd synonym for ufs2 - - sun used in SunOS (Solaris) - supported as read-write - - sunx86 used in SunOS for Intel (Solarisx86) - supported as read-write - - hp used in HP-UX - supported as read-only - - nextstep - used in NextStep - supported as read-only - - nextstep-cd - used for NextStep CDROMs (block_size == 2048) - supported as read-only - - openstep - used in OpenStep - supported as read-only - - -POSSIBLE PROBLEMS -================= - -See next section, if you have any. - - -BUG REPORTS -=========== - -Any ufs bug report you can send to daniel.pirkl@email.cz or -to dushistov@mail.ru (do not send partition tables bug reports). diff --git a/Documentation/filesystems/vfs.rst b/Documentation/filesystems/vfs.rst new file mode 100644 index 000000000000..7d4d09dd5e6d --- /dev/null +++ b/Documentation/filesystems/vfs.rst @@ -0,0 +1,1428 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================================= +Overview of the Linux Virtual File System +========================================= + +Original author: Richard Gooch <rgooch@atnf.csiro.au> + +- Copyright (C) 1999 Richard Gooch +- Copyright (C) 2005 Pekka Enberg + + +Introduction +============ + +The Virtual File System (also known as the Virtual Filesystem Switch) is +the software layer in the kernel that provides the filesystem interface +to userspace programs. It also provides an abstraction within the +kernel which allows different filesystem implementations to coexist. + +VFS system calls open(2), stat(2), read(2), write(2), chmod(2) and so on +are called from a process context. Filesystem locking is described in +the document Documentation/filesystems/locking.rst. + + +Directory Entry Cache (dcache) +------------------------------ + +The VFS implements the open(2), stat(2), chmod(2), and similar system +calls. The pathname argument that is passed to them is used by the VFS +to search through the directory entry cache (also known as the dentry +cache or dcache). This provides a very fast look-up mechanism to +translate a pathname (filename) into a specific dentry. Dentries live +in RAM and are never saved to disc: they exist only for performance. + +The dentry cache is meant to be a view into your entire filespace. As +most computers cannot fit all dentries in the RAM at the same time, some +bits of the cache are missing. In order to resolve your pathname into a +dentry, the VFS may have to resort to creating dentries along the way, +and then loading the inode. This is done by looking up the inode. + + +The Inode Object +---------------- + +An individual dentry usually has a pointer to an inode. Inodes are +filesystem objects such as regular files, directories, FIFOs and other +beasts. They live either on the disc (for block device filesystems) or +in the memory (for pseudo filesystems). Inodes that live on the disc +are copied into the memory when required and changes to the inode are +written back to disc. A single inode can be pointed to by multiple +dentries (hard links, for example, do this). + +To look up an inode requires that the VFS calls the lookup() method of +the parent directory inode. This method is installed by the specific +filesystem implementation that the inode lives in. Once the VFS has the +required dentry (and hence the inode), we can do all those boring things +like open(2) the file, or stat(2) it to peek at the inode data. The +stat(2) operation is fairly simple: once the VFS has the dentry, it +peeks at the inode data and passes some of it back to userspace. + + +The File Object +--------------- + +Opening a file requires another operation: allocation of a file +structure (this is the kernel-side implementation of file descriptors). +The freshly allocated file structure is initialized with a pointer to +the dentry and a set of file operation member functions. These are +taken from the inode data. The open() file method is then called so the +specific filesystem implementation can do its work. You can see that +this is another switch performed by the VFS. The file structure is +placed into the file descriptor table for the process. + +Reading, writing and closing files (and other assorted VFS operations) +is done by using the userspace file descriptor to grab the appropriate +file structure, and then calling the required file structure method to +do whatever is required. For as long as the file is open, it keeps the +dentry in use, which in turn means that the VFS inode is still in use. + + +Registering and Mounting a Filesystem +===================================== + +To register and unregister a filesystem, use the following API +functions: + +.. code-block:: c + + #include <linux/fs.h> + + extern int register_filesystem(struct file_system_type *); + extern int unregister_filesystem(struct file_system_type *); + +The passed struct file_system_type describes your filesystem. When a +request is made to mount a filesystem onto a directory in your +namespace, the VFS will call the appropriate mount() method for the +specific filesystem. New vfsmount referring to the tree returned by +->mount() will be attached to the mountpoint, so that when pathname +resolution reaches the mountpoint it will jump into the root of that +vfsmount. + +You can see all filesystems that are registered to the kernel in the +file /proc/filesystems. + + +struct file_system_type +----------------------- + +This describes the filesystem. As of kernel 2.6.39, the following +members are defined: + +.. code-block:: c + + struct file_system_operations { + const char *name; + int fs_flags; + struct dentry *(*mount) (struct file_system_type *, int, + const char *, void *); + void (*kill_sb) (struct super_block *); + struct module *owner; + struct file_system_type * next; + struct list_head fs_supers; + struct lock_class_key s_lock_key; + struct lock_class_key s_umount_key; + }; + +``name`` + the name of the filesystem type, such as "ext2", "iso9660", + "msdos" and so on + +``fs_flags`` + various flags (i.e. FS_REQUIRES_DEV, FS_NO_DCACHE, etc.) + +``mount`` + the method to call when a new instance of this filesystem should + be mounted + +``kill_sb`` + the method to call when an instance of this filesystem should be + shut down + + +``owner`` + for internal VFS use: you should initialize this to THIS_MODULE + in most cases. + +``next`` + for internal VFS use: you should initialize this to NULL + + s_lock_key, s_umount_key: lockdep-specific + +The mount() method has the following arguments: + +``struct file_system_type *fs_type`` + describes the filesystem, partly initialized by the specific + filesystem code + +``int flags`` + mount flags + +``const char *dev_name`` + the device name we are mounting. + +``void *data`` + arbitrary mount options, usually comes as an ASCII string (see + "Mount Options" section) + +The mount() method must return the root dentry of the tree requested by +caller. An active reference to its superblock must be grabbed and the +superblock must be locked. On failure it should return ERR_PTR(error). + +The arguments match those of mount(2) and their interpretation depends +on filesystem type. E.g. for block filesystems, dev_name is interpreted +as block device name, that device is opened and if it contains a +suitable filesystem image the method creates and initializes struct +super_block accordingly, returning its root dentry to caller. + +->mount() may choose to return a subtree of existing filesystem - it +doesn't have to create a new one. The main result from the caller's +point of view is a reference to dentry at the root of (sub)tree to be +attached; creation of new superblock is a common side effect. + +The most interesting member of the superblock structure that the mount() +method fills in is the "s_op" field. This is a pointer to a "struct +super_operations" which describes the next level of the filesystem +implementation. + +Usually, a filesystem uses one of the generic mount() implementations +and provides a fill_super() callback instead. The generic variants are: + +``mount_bdev`` + mount a filesystem residing on a block device + +``mount_nodev`` + mount a filesystem that is not backed by a device + +``mount_single`` + mount a filesystem which shares the instance between all mounts + +A fill_super() callback implementation has the following arguments: + +``struct super_block *sb`` + the superblock structure. The callback must initialize this + properly. + +``void *data`` + arbitrary mount options, usually comes as an ASCII string (see + "Mount Options" section) + +``int silent`` + whether or not to be silent on error + + +The Superblock Object +===================== + +A superblock object represents a mounted filesystem. + + +struct super_operations +----------------------- + +This describes how the VFS can manipulate the superblock of your +filesystem. As of kernel 2.6.22, the following members are defined: + +.. code-block:: c + + struct super_operations { + struct inode *(*alloc_inode)(struct super_block *sb); + void (*destroy_inode)(struct inode *); + + void (*dirty_inode) (struct inode *, int flags); + int (*write_inode) (struct inode *, int); + void (*drop_inode) (struct inode *); + void (*delete_inode) (struct inode *); + void (*put_super) (struct super_block *); + int (*sync_fs)(struct super_block *sb, int wait); + int (*freeze_fs) (struct super_block *); + int (*unfreeze_fs) (struct super_block *); + int (*statfs) (struct dentry *, struct kstatfs *); + int (*remount_fs) (struct super_block *, int *, char *); + void (*clear_inode) (struct inode *); + void (*umount_begin) (struct super_block *); + + int (*show_options)(struct seq_file *, struct dentry *); + + ssize_t (*quota_read)(struct super_block *, int, char *, size_t, loff_t); + ssize_t (*quota_write)(struct super_block *, int, const char *, size_t, loff_t); + int (*nr_cached_objects)(struct super_block *); + void (*free_cached_objects)(struct super_block *, int); + }; + +All methods are called without any locks being held, unless otherwise +noted. This means that most methods can block safely. All methods are +only called from a process context (i.e. not from an interrupt handler +or bottom half). + +``alloc_inode`` + this method is called by alloc_inode() to allocate memory for + struct inode and initialize it. If this function is not + defined, a simple 'struct inode' is allocated. Normally + alloc_inode will be used to allocate a larger structure which + contains a 'struct inode' embedded within it. + +``destroy_inode`` + this method is called by destroy_inode() to release resources + allocated for struct inode. It is only required if + ->alloc_inode was defined and simply undoes anything done by + ->alloc_inode. + +``dirty_inode`` + this method is called by the VFS to mark an inode dirty. + +``write_inode`` + this method is called when the VFS needs to write an inode to + disc. The second parameter indicates whether the write should + be synchronous or not, not all filesystems check this flag. + +``drop_inode`` + called when the last access to the inode is dropped, with the + inode->i_lock spinlock held. + + This method should be either NULL (normal UNIX filesystem + semantics) or "generic_delete_inode" (for filesystems that do + not want to cache inodes - causing "delete_inode" to always be + called regardless of the value of i_nlink) + + The "generic_delete_inode()" behavior is equivalent to the old + practice of using "force_delete" in the put_inode() case, but + does not have the races that the "force_delete()" approach had. + +``delete_inode`` + called when the VFS wants to delete an inode + +``put_super`` + called when the VFS wishes to free the superblock + (i.e. unmount). This is called with the superblock lock held + +``sync_fs`` + called when VFS is writing out all dirty data associated with a + superblock. The second parameter indicates whether the method + should wait until the write out has been completed. Optional. + +``freeze_fs`` + called when VFS is locking a filesystem and forcing it into a + consistent state. This method is currently used by the Logical + Volume Manager (LVM). + +``unfreeze_fs`` + called when VFS is unlocking a filesystem and making it writable + again. + +``statfs`` + called when the VFS needs to get filesystem statistics. + +``remount_fs`` + called when the filesystem is remounted. This is called with + the kernel lock held + +``clear_inode`` + called then the VFS clears the inode. Optional + +``umount_begin`` + called when the VFS is unmounting a filesystem. + +``show_options`` + called by the VFS to show mount options for /proc/<pid>/mounts. + (see "Mount Options" section) + +``quota_read`` + called by the VFS to read from filesystem quota file. + +``quota_write`` + called by the VFS to write to filesystem quota file. + +``nr_cached_objects`` + called by the sb cache shrinking function for the filesystem to + return the number of freeable cached objects it contains. + Optional. + +``free_cache_objects`` + called by the sb cache shrinking function for the filesystem to + scan the number of objects indicated to try to free them. + Optional, but any filesystem implementing this method needs to + also implement ->nr_cached_objects for it to be called + correctly. + + We can't do anything with any errors that the filesystem might + encountered, hence the void return type. This will never be + called if the VM is trying to reclaim under GFP_NOFS conditions, + hence this method does not need to handle that situation itself. + + Implementations must include conditional reschedule calls inside + any scanning loop that is done. This allows the VFS to + determine appropriate scan batch sizes without having to worry + about whether implementations will cause holdoff problems due to + large scan batch sizes. + +Whoever sets up the inode is responsible for filling in the "i_op" +field. This is a pointer to a "struct inode_operations" which describes +the methods that can be performed on individual inodes. + + +struct xattr_handlers +--------------------- + +On filesystems that support extended attributes (xattrs), the s_xattr +superblock field points to a NULL-terminated array of xattr handlers. +Extended attributes are name:value pairs. + +``name`` + Indicates that the handler matches attributes with the specified + name (such as "system.posix_acl_access"); the prefix field must + be NULL. + +``prefix`` + Indicates that the handler matches all attributes with the + specified name prefix (such as "user."); the name field must be + NULL. + +``list`` + Determine if attributes matching this xattr handler should be + listed for a particular dentry. Used by some listxattr + implementations like generic_listxattr. + +``get`` + Called by the VFS to get the value of a particular extended + attribute. This method is called by the getxattr(2) system + call. + +``set`` + Called by the VFS to set the value of a particular extended + attribute. When the new value is NULL, called to remove a + particular extended attribute. This method is called by the the + setxattr(2) and removexattr(2) system calls. + +When none of the xattr handlers of a filesystem match the specified +attribute name or when a filesystem doesn't support extended attributes, +the various ``*xattr(2)`` system calls return -EOPNOTSUPP. + + +The Inode Object +================ + +An inode object represents an object within the filesystem. + + +struct inode_operations +----------------------- + +This describes how the VFS can manipulate an inode in your filesystem. +As of kernel 2.6.22, the following members are defined: + +.. code-block:: c + + struct inode_operations { + int (*create) (struct inode *,struct dentry *, umode_t, bool); + struct dentry * (*lookup) (struct inode *,struct dentry *, unsigned int); + int (*link) (struct dentry *,struct inode *,struct dentry *); + int (*unlink) (struct inode *,struct dentry *); + int (*symlink) (struct inode *,struct dentry *,const char *); + int (*mkdir) (struct inode *,struct dentry *,umode_t); + int (*rmdir) (struct inode *,struct dentry *); + int (*mknod) (struct inode *,struct dentry *,umode_t,dev_t); + int (*rename) (struct inode *, struct dentry *, + struct inode *, struct dentry *, unsigned int); + int (*readlink) (struct dentry *, char __user *,int); + const char *(*get_link) (struct dentry *, struct inode *, + struct delayed_call *); + int (*permission) (struct inode *, int); + int (*get_acl)(struct inode *, int); + int (*setattr) (struct dentry *, struct iattr *); + int (*getattr) (const struct path *, struct kstat *, u32, unsigned int); + ssize_t (*listxattr) (struct dentry *, char *, size_t); + void (*update_time)(struct inode *, struct timespec *, int); + int (*atomic_open)(struct inode *, struct dentry *, struct file *, + unsigned open_flag, umode_t create_mode); + int (*tmpfile) (struct inode *, struct dentry *, umode_t); + }; + +Again, all methods are called without any locks being held, unless +otherwise noted. + +``create`` + called by the open(2) and creat(2) system calls. Only required + if you want to support regular files. The dentry you get should + not have an inode (i.e. it should be a negative dentry). Here + you will probably call d_instantiate() with the dentry and the + newly created inode + +``lookup`` + called when the VFS needs to look up an inode in a parent + directory. The name to look for is found in the dentry. This + method must call d_add() to insert the found inode into the + dentry. The "i_count" field in the inode structure should be + incremented. If the named inode does not exist a NULL inode + should be inserted into the dentry (this is called a negative + dentry). Returning an error code from this routine must only be + done on a real error, otherwise creating inodes with system + calls like create(2), mknod(2), mkdir(2) and so on will fail. + If you wish to overload the dentry methods then you should + initialise the "d_dop" field in the dentry; this is a pointer to + a struct "dentry_operations". This method is called with the + directory inode semaphore held + +``link`` + called by the link(2) system call. Only required if you want to + support hard links. You will probably need to call + d_instantiate() just as you would in the create() method + +``unlink`` + called by the unlink(2) system call. Only required if you want + to support deleting inodes + +``symlink`` + called by the symlink(2) system call. Only required if you want + to support symlinks. You will probably need to call + d_instantiate() just as you would in the create() method + +``mkdir`` + called by the mkdir(2) system call. Only required if you want + to support creating subdirectories. You will probably need to + call d_instantiate() just as you would in the create() method + +``rmdir`` + called by the rmdir(2) system call. Only required if you want + to support deleting subdirectories + +``mknod`` + called by the mknod(2) system call to create a device (char, + block) inode or a named pipe (FIFO) or socket. Only required if + you want to support creating these types of inodes. You will + probably need to call d_instantiate() just as you would in the + create() method + +``rename`` + called by the rename(2) system call to rename the object to have + the parent and name given by the second inode and dentry. + + The filesystem must return -EINVAL for any unsupported or + unknown flags. Currently the following flags are implemented: + (1) RENAME_NOREPLACE: this flag indicates that if the target of + the rename exists the rename should fail with -EEXIST instead of + replacing the target. The VFS already checks for existence, so + for local filesystems the RENAME_NOREPLACE implementation is + equivalent to plain rename. + (2) RENAME_EXCHANGE: exchange source and target. Both must + exist; this is checked by the VFS. Unlike plain rename, source + and target may be of different type. + +``get_link`` + called by the VFS to follow a symbolic link to the inode it + points to. Only required if you want to support symbolic links. + This method returns the symlink body to traverse (and possibly + resets the current position with nd_jump_link()). If the body + won't go away until the inode is gone, nothing else is needed; + if it needs to be otherwise pinned, arrange for its release by + having get_link(..., ..., done) do set_delayed_call(done, + destructor, argument). In that case destructor(argument) will + be called once VFS is done with the body you've returned. May + be called in RCU mode; that is indicated by NULL dentry + argument. If request can't be handled without leaving RCU mode, + have it return ERR_PTR(-ECHILD). + + If the filesystem stores the symlink target in ->i_link, the + VFS may use it directly without calling ->get_link(); however, + ->get_link() must still be provided. ->i_link must not be + freed until after an RCU grace period. Writing to ->i_link + post-iget() time requires a 'release' memory barrier. + +``readlink`` + this is now just an override for use by readlink(2) for the + cases when ->get_link uses nd_jump_link() or object is not in + fact a symlink. Normally filesystems should only implement + ->get_link for symlinks and readlink(2) will automatically use + that. + +``permission`` + called by the VFS to check for access rights on a POSIX-like + filesystem. + + May be called in rcu-walk mode (mask & MAY_NOT_BLOCK). If in + rcu-walk mode, the filesystem must check the permission without + blocking or storing to the inode. + + If a situation is encountered that rcu-walk cannot handle, + return + -ECHILD and it will be called again in ref-walk mode. + +``setattr`` + called by the VFS to set attributes for a file. This method is + called by chmod(2) and related system calls. + +``getattr`` + called by the VFS to get attributes of a file. This method is + called by stat(2) and related system calls. + +``listxattr`` + called by the VFS to list all extended attributes for a given + file. This method is called by the listxattr(2) system call. + +``update_time`` + called by the VFS to update a specific time or the i_version of + an inode. If this is not defined the VFS will update the inode + itself and call mark_inode_dirty_sync. + +``atomic_open`` + called on the last component of an open. Using this optional + method the filesystem can look up, possibly create and open the + file in one atomic operation. If it wants to leave actual + opening to the caller (e.g. if the file turned out to be a + symlink, device, or just something filesystem won't do atomic + open for), it may signal this by returning finish_no_open(file, + dentry). This method is only called if the last component is + negative or needs lookup. Cached positive dentries are still + handled by f_op->open(). If the file was created, FMODE_CREATED + flag should be set in file->f_mode. In case of O_EXCL the + method must only succeed if the file didn't exist and hence + FMODE_CREATED shall always be set on success. + +``tmpfile`` + called in the end of O_TMPFILE open(). Optional, equivalent to + atomically creating, opening and unlinking a file in given + directory. + + +The Address Space Object +======================== + +The address space object is used to group and manage pages in the page +cache. It can be used to keep track of the pages in a file (or anything +else) and also track the mapping of sections of the file into process +address spaces. + +There are a number of distinct yet related services that an +address-space can provide. These include communicating memory pressure, +page lookup by address, and keeping track of pages tagged as Dirty or +Writeback. + +The first can be used independently to the others. The VM can try to +either write dirty pages in order to clean them, or release clean pages +in order to reuse them. To do this it can call the ->writepage method +on dirty pages, and ->releasepage on clean pages with PagePrivate set. +Clean pages without PagePrivate and with no external references will be +released without notice being given to the address_space. + +To achieve this functionality, pages need to be placed on an LRU with +lru_cache_add and mark_page_active needs to be called whenever the page +is used. + +Pages are normally kept in a radix tree index by ->index. This tree +maintains information about the PG_Dirty and PG_Writeback status of each +page, so that pages with either of these flags can be found quickly. + +The Dirty tag is primarily used by mpage_writepages - the default +->writepages method. It uses the tag to find dirty pages to call +->writepage on. If mpage_writepages is not used (i.e. the address +provides its own ->writepages) , the PAGECACHE_TAG_DIRTY tag is almost +unused. write_inode_now and sync_inode do use it (through +__sync_single_inode) to check if ->writepages has been successful in +writing out the whole address_space. + +The Writeback tag is used by filemap*wait* and sync_page* functions, via +filemap_fdatawait_range, to wait for all writeback to complete. + +An address_space handler may attach extra information to a page, +typically using the 'private' field in the 'struct page'. If such +information is attached, the PG_Private flag should be set. This will +cause various VM routines to make extra calls into the address_space +handler to deal with that data. + +An address space acts as an intermediate between storage and +application. Data is read into the address space a whole page at a +time, and provided to the application either by copying of the page, or +by memory-mapping the page. Data is written into the address space by +the application, and then written-back to storage typically in whole +pages, however the address_space has finer control of write sizes. + +The read process essentially only requires 'readpage'. The write +process is more complicated and uses write_begin/write_end or +set_page_dirty to write data into the address_space, and writepage and +writepages to writeback data to storage. + +Adding and removing pages to/from an address_space is protected by the +inode's i_mutex. + +When data is written to a page, the PG_Dirty flag should be set. It +typically remains set until writepage asks for it to be written. This +should clear PG_Dirty and set PG_Writeback. It can be actually written +at any point after PG_Dirty is clear. Once it is known to be safe, +PG_Writeback is cleared. + +Writeback makes use of a writeback_control structure to direct the +operations. This gives the the writepage and writepages operations some +information about the nature of and reason for the writeback request, +and the constraints under which it is being done. It is also used to +return information back to the caller about the result of a writepage or +writepages request. + + +Handling errors during writeback +-------------------------------- + +Most applications that do buffered I/O will periodically call a file +synchronization call (fsync, fdatasync, msync or sync_file_range) to +ensure that data written has made it to the backing store. When there +is an error during writeback, they expect that error to be reported when +a file sync request is made. After an error has been reported on one +request, subsequent requests on the same file descriptor should return +0, unless further writeback errors have occurred since the previous file +syncronization. + +Ideally, the kernel would report errors only on file descriptions on +which writes were done that subsequently failed to be written back. The +generic pagecache infrastructure does not track the file descriptions +that have dirtied each individual page however, so determining which +file descriptors should get back an error is not possible. + +Instead, the generic writeback error tracking infrastructure in the +kernel settles for reporting errors to fsync on all file descriptions +that were open at the time that the error occurred. In a situation with +multiple writers, all of them will get back an error on a subsequent +fsync, even if all of the writes done through that particular file +descriptor succeeded (or even if there were no writes on that file +descriptor at all). + +Filesystems that wish to use this infrastructure should call +mapping_set_error to record the error in the address_space when it +occurs. Then, after writing back data from the pagecache in their +file->fsync operation, they should call file_check_and_advance_wb_err to +ensure that the struct file's error cursor has advanced to the correct +point in the stream of errors emitted by the backing device(s). + + +struct address_space_operations +------------------------------- + +This describes how the VFS can manipulate mapping of a file to page +cache in your filesystem. The following members are defined: + +.. code-block:: c + + struct address_space_operations { + int (*writepage)(struct page *page, struct writeback_control *wbc); + int (*readpage)(struct file *, struct page *); + int (*writepages)(struct address_space *, struct writeback_control *); + int (*set_page_dirty)(struct page *page); + int (*readpages)(struct file *filp, struct address_space *mapping, + struct list_head *pages, unsigned nr_pages); + int (*write_begin)(struct file *, struct address_space *mapping, + loff_t pos, unsigned len, unsigned flags, + struct page **pagep, void **fsdata); + int (*write_end)(struct file *, struct address_space *mapping, + loff_t pos, unsigned len, unsigned copied, + struct page *page, void *fsdata); + sector_t (*bmap)(struct address_space *, sector_t); + void (*invalidatepage) (struct page *, unsigned int, unsigned int); + int (*releasepage) (struct page *, int); + void (*freepage)(struct page *); + ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter); + /* isolate a page for migration */ + bool (*isolate_page) (struct page *, isolate_mode_t); + /* migrate the contents of a page to the specified target */ + int (*migratepage) (struct page *, struct page *); + /* put migration-failed page back to right list */ + void (*putback_page) (struct page *); + int (*launder_page) (struct page *); + + int (*is_partially_uptodate) (struct page *, unsigned long, + unsigned long); + void (*is_dirty_writeback) (struct page *, bool *, bool *); + int (*error_remove_page) (struct mapping *mapping, struct page *page); + int (*swap_activate)(struct file *); + int (*swap_deactivate)(struct file *); + }; + +``writepage`` + called by the VM to write a dirty page to backing store. This + may happen for data integrity reasons (i.e. 'sync'), or to free + up memory (flush). The difference can be seen in + wbc->sync_mode. The PG_Dirty flag has been cleared and + PageLocked is true. writepage should start writeout, should set + PG_Writeback, and should make sure the page is unlocked, either + synchronously or asynchronously when the write operation + completes. + + If wbc->sync_mode is WB_SYNC_NONE, ->writepage doesn't have to + try too hard if there are problems, and may choose to write out + other pages from the mapping if that is easier (e.g. due to + internal dependencies). If it chooses not to start writeout, it + should return AOP_WRITEPAGE_ACTIVATE so that the VM will not + keep calling ->writepage on that page. + + See the file "Locking" for more details. + +``readpage`` + called by the VM to read a page from backing store. The page + will be Locked when readpage is called, and should be unlocked + and marked uptodate once the read completes. If ->readpage + discovers that it needs to unlock the page for some reason, it + can do so, and then return AOP_TRUNCATED_PAGE. In this case, + the page will be relocated, relocked and if that all succeeds, + ->readpage will be called again. + +``writepages`` + called by the VM to write out pages associated with the + address_space object. If wbc->sync_mode is WBC_SYNC_ALL, then + the writeback_control will specify a range of pages that must be + written out. If it is WBC_SYNC_NONE, then a nr_to_write is + given and that many pages should be written if possible. If no + ->writepages is given, then mpage_writepages is used instead. + This will choose pages from the address space that are tagged as + DIRTY and will pass them to ->writepage. + +``set_page_dirty`` + called by the VM to set a page dirty. This is particularly + needed if an address space attaches private data to a page, and + that data needs to be updated when a page is dirtied. This is + called, for example, when a memory mapped page gets modified. + If defined, it should set the PageDirty flag, and the + PAGECACHE_TAG_DIRTY tag in the radix tree. + +``readpages`` + called by the VM to read pages associated with the address_space + object. This is essentially just a vector version of readpage. + Instead of just one page, several pages are requested. + readpages is only used for read-ahead, so read errors are + ignored. If anything goes wrong, feel free to give up. + +``write_begin`` + Called by the generic buffered write code to ask the filesystem + to prepare to write len bytes at the given offset in the file. + The address_space should check that the write will be able to + complete, by allocating space if necessary and doing any other + internal housekeeping. If the write will update parts of any + basic-blocks on storage, then those blocks should be pre-read + (if they haven't been read already) so that the updated blocks + can be written out properly. + + The filesystem must return the locked pagecache page for the + specified offset, in ``*pagep``, for the caller to write into. + + It must be able to cope with short writes (where the length + passed to write_begin is greater than the number of bytes copied + into the page). + + flags is a field for AOP_FLAG_xxx flags, described in + include/linux/fs.h. + + A void * may be returned in fsdata, which then gets passed into + write_end. + + Returns 0 on success; < 0 on failure (which is the error code), + in which case write_end is not called. + +``write_end`` + After a successful write_begin, and data copy, write_end must be + called. len is the original len passed to write_begin, and + copied is the amount that was able to be copied. + + The filesystem must take care of unlocking the page and + releasing it refcount, and updating i_size. + + Returns < 0 on failure, otherwise the number of bytes (<= + 'copied') that were able to be copied into pagecache. + +``bmap`` + called by the VFS to map a logical block offset within object to + physical block number. This method is used by the FIBMAP ioctl + and for working with swap-files. To be able to swap to a file, + the file must have a stable mapping to a block device. The swap + system does not go through the filesystem but instead uses bmap + to find out where the blocks in the file are and uses those + addresses directly. + +``invalidatepage`` + If a page has PagePrivate set, then invalidatepage will be + called when part or all of the page is to be removed from the + address space. This generally corresponds to either a + truncation, punch hole or a complete invalidation of the address + space (in the latter case 'offset' will always be 0 and 'length' + will be PAGE_SIZE). Any private data associated with the page + should be updated to reflect this truncation. If offset is 0 + and length is PAGE_SIZE, then the private data should be + released, because the page must be able to be completely + discarded. This may be done by calling the ->releasepage + function, but in this case the release MUST succeed. + +``releasepage`` + releasepage is called on PagePrivate pages to indicate that the + page should be freed if possible. ->releasepage should remove + any private data from the page and clear the PagePrivate flag. + If releasepage() fails for some reason, it must indicate failure + with a 0 return value. releasepage() is used in two distinct + though related cases. The first is when the VM finds a clean + page with no active users and wants to make it a free page. If + ->releasepage succeeds, the page will be removed from the + address_space and become free. + + The second case is when a request has been made to invalidate + some or all pages in an address_space. This can happen through + the fadvise(POSIX_FADV_DONTNEED) system call or by the + filesystem explicitly requesting it as nfs and 9fs do (when they + believe the cache may be out of date with storage) by calling + invalidate_inode_pages2(). If the filesystem makes such a call, + and needs to be certain that all pages are invalidated, then its + releasepage will need to ensure this. Possibly it can clear the + PageUptodate bit if it cannot free private data yet. + +``freepage`` + freepage is called once the page is no longer visible in the + page cache in order to allow the cleanup of any private data. + Since it may be called by the memory reclaimer, it should not + assume that the original address_space mapping still exists, and + it should not block. + +``direct_IO`` + called by the generic read/write routines to perform direct_IO - + that is IO requests which bypass the page cache and transfer + data directly between the storage and the application's address + space. + +``isolate_page`` + Called by the VM when isolating a movable non-lru page. If page + is successfully isolated, VM marks the page as PG_isolated via + __SetPageIsolated. + +``migrate_page`` + This is used to compact the physical memory usage. If the VM + wants to relocate a page (maybe off a memory card that is + signalling imminent failure) it will pass a new page and an old + page to this function. migrate_page should transfer any private + data across and update any references that it has to the page. + +``putback_page`` + Called by the VM when isolated page's migration fails. + +``launder_page`` + Called before freeing a page - it writes back the dirty page. + To prevent redirtying the page, it is kept locked during the + whole operation. + +``is_partially_uptodate`` + Called by the VM when reading a file through the pagecache when + the underlying blocksize != pagesize. If the required block is + up to date then the read can complete without needing the IO to + bring the whole page up to date. + +``is_dirty_writeback`` + Called by the VM when attempting to reclaim a page. The VM uses + dirty and writeback information to determine if it needs to + stall to allow flushers a chance to complete some IO. + Ordinarily it can use PageDirty and PageWriteback but some + filesystems have more complex state (unstable pages in NFS + prevent reclaim) or do not set those flags due to locking + problems. This callback allows a filesystem to indicate to the + VM if a page should be treated as dirty or writeback for the + purposes of stalling. + +``error_remove_page`` + normally set to generic_error_remove_page if truncation is ok + for this address space. Used for memory failure handling. + Setting this implies you deal with pages going away under you, + unless you have them locked or reference counts increased. + +``swap_activate`` + Called when swapon is used on a file to allocate space if + necessary and pin the block lookup information in memory. A + return value of zero indicates success, in which case this file + can be used to back swapspace. + +``swap_deactivate`` + Called during swapoff on files where swap_activate was + successful. + + +The File Object +=============== + +A file object represents a file opened by a process. This is also known +as an "open file description" in POSIX parlance. + + +struct file_operations +---------------------- + +This describes how the VFS can manipulate an open file. As of kernel +4.18, the following members are defined: + +.. code-block:: c + + struct file_operations { + struct module *owner; + loff_t (*llseek) (struct file *, loff_t, int); + ssize_t (*read) (struct file *, char __user *, size_t, loff_t *); + ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); + ssize_t (*read_iter) (struct kiocb *, struct iov_iter *); + ssize_t (*write_iter) (struct kiocb *, struct iov_iter *); + int (*iopoll)(struct kiocb *kiocb, bool spin); + int (*iterate) (struct file *, struct dir_context *); + int (*iterate_shared) (struct file *, struct dir_context *); + __poll_t (*poll) (struct file *, struct poll_table_struct *); + long (*unlocked_ioctl) (struct file *, unsigned int, unsigned long); + long (*compat_ioctl) (struct file *, unsigned int, unsigned long); + int (*mmap) (struct file *, struct vm_area_struct *); + int (*open) (struct inode *, struct file *); + int (*flush) (struct file *, fl_owner_t id); + int (*release) (struct inode *, struct file *); + int (*fsync) (struct file *, loff_t, loff_t, int datasync); + int (*fasync) (int, struct file *, int); + int (*lock) (struct file *, int, struct file_lock *); + ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int); + unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long); + int (*check_flags)(int); + int (*flock) (struct file *, int, struct file_lock *); + ssize_t (*splice_write)(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int); + ssize_t (*splice_read)(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); + int (*setlease)(struct file *, long, struct file_lock **, void **); + long (*fallocate)(struct file *file, int mode, loff_t offset, + loff_t len); + void (*show_fdinfo)(struct seq_file *m, struct file *f); + #ifndef CONFIG_MMU + unsigned (*mmap_capabilities)(struct file *); + #endif + ssize_t (*copy_file_range)(struct file *, loff_t, struct file *, loff_t, size_t, unsigned int); + loff_t (*remap_file_range)(struct file *file_in, loff_t pos_in, + struct file *file_out, loff_t pos_out, + loff_t len, unsigned int remap_flags); + int (*fadvise)(struct file *, loff_t, loff_t, int); + }; + +Again, all methods are called without any locks being held, unless +otherwise noted. + +``llseek`` + called when the VFS needs to move the file position index + +``read`` + called by read(2) and related system calls + +``read_iter`` + possibly asynchronous read with iov_iter as destination + +``write`` + called by write(2) and related system calls + +``write_iter`` + possibly asynchronous write with iov_iter as source + +``iopoll`` + called when aio wants to poll for completions on HIPRI iocbs + +``iterate`` + called when the VFS needs to read the directory contents + +``iterate_shared`` + called when the VFS needs to read the directory contents when + filesystem supports concurrent dir iterators + +``poll`` + called by the VFS when a process wants to check if there is + activity on this file and (optionally) go to sleep until there + is activity. Called by the select(2) and poll(2) system calls + +``unlocked_ioctl`` + called by the ioctl(2) system call. + +``compat_ioctl`` + called by the ioctl(2) system call when 32 bit system calls are + used on 64 bit kernels. + +``mmap`` + called by the mmap(2) system call + +``open`` + called by the VFS when an inode should be opened. When the VFS + opens a file, it creates a new "struct file". It then calls the + open method for the newly allocated file structure. You might + think that the open method really belongs in "struct + inode_operations", and you may be right. I think it's done the + way it is because it makes filesystems simpler to implement. + The open() method is a good place to initialize the + "private_data" member in the file structure if you want to point + to a device structure + +``flush`` + called by the close(2) system call to flush a file + +``release`` + called when the last reference to an open file is closed + +``fsync`` + called by the fsync(2) system call. Also see the section above + entitled "Handling errors during writeback". + +``fasync`` + called by the fcntl(2) system call when asynchronous + (non-blocking) mode is enabled for a file + +``lock`` + called by the fcntl(2) system call for F_GETLK, F_SETLK, and + F_SETLKW commands + +``get_unmapped_area`` + called by the mmap(2) system call + +``check_flags`` + called by the fcntl(2) system call for F_SETFL command + +``flock`` + called by the flock(2) system call + +``splice_write`` + called by the VFS to splice data from a pipe to a file. This + method is used by the splice(2) system call + +``splice_read`` + called by the VFS to splice data from file to a pipe. This + method is used by the splice(2) system call + +``setlease`` + called by the VFS to set or release a file lock lease. setlease + implementations should call generic_setlease to record or remove + the lease in the inode after setting it. + +``fallocate`` + called by the VFS to preallocate blocks or punch a hole. + +``copy_file_range`` + called by the copy_file_range(2) system call. + +``remap_file_range`` + called by the ioctl(2) system call for FICLONERANGE and FICLONE + and FIDEDUPERANGE commands to remap file ranges. An + implementation should remap len bytes at pos_in of the source + file into the dest file at pos_out. Implementations must handle + callers passing in len == 0; this means "remap to the end of the + source file". The return value should the number of bytes + remapped, or the usual negative error code if errors occurred + before any bytes were remapped. The remap_flags parameter + accepts REMAP_FILE_* flags. If REMAP_FILE_DEDUP is set then the + implementation must only remap if the requested file ranges have + identical contents. If REMAP_CAN_SHORTEN is set, the caller is + ok with the implementation shortening the request length to + satisfy alignment or EOF requirements (or any other reason). + +``fadvise`` + possibly called by the fadvise64() system call. + +Note that the file operations are implemented by the specific +filesystem in which the inode resides. When opening a device node +(character or block special) most filesystems will call special +support routines in the VFS which will locate the required device +driver information. These support routines replace the filesystem file +operations with those for the device driver, and then proceed to call +the new open() method for the file. This is how opening a device file +in the filesystem eventually ends up calling the device driver open() +method. + + +Directory Entry Cache (dcache) +============================== + + +struct dentry_operations +------------------------ + +This describes how a filesystem can overload the standard dentry +operations. Dentries and the dcache are the domain of the VFS and the +individual filesystem implementations. Device drivers have no business +here. These methods may be set to NULL, as they are either optional or +the VFS uses a default. As of kernel 2.6.22, the following members are +defined: + +.. code-block:: c + + struct dentry_operations { + int (*d_revalidate)(struct dentry *, unsigned int); + int (*d_weak_revalidate)(struct dentry *, unsigned int); + int (*d_hash)(const struct dentry *, struct qstr *); + int (*d_compare)(const struct dentry *, + unsigned int, const char *, const struct qstr *); + int (*d_delete)(const struct dentry *); + int (*d_init)(struct dentry *); + void (*d_release)(struct dentry *); + void (*d_iput)(struct dentry *, struct inode *); + char *(*d_dname)(struct dentry *, char *, int); + struct vfsmount *(*d_automount)(struct path *); + int (*d_manage)(const struct path *, bool); + struct dentry *(*d_real)(struct dentry *, const struct inode *); + }; + +``d_revalidate`` + called when the VFS needs to revalidate a dentry. This is + called whenever a name look-up finds a dentry in the dcache. + Most local filesystems leave this as NULL, because all their + dentries in the dcache are valid. Network filesystems are + different since things can change on the server without the + client necessarily being aware of it. + + This function should return a positive value if the dentry is + still valid, and zero or a negative error code if it isn't. + + d_revalidate may be called in rcu-walk mode (flags & + LOOKUP_RCU). If in rcu-walk mode, the filesystem must + revalidate the dentry without blocking or storing to the dentry, + d_parent and d_inode should not be used without care (because + they can change and, in d_inode case, even become NULL under + us). + + If a situation is encountered that rcu-walk cannot handle, + return + -ECHILD and it will be called again in ref-walk mode. + +``_weak_revalidate`` + called when the VFS needs to revalidate a "jumped" dentry. This + is called when a path-walk ends at dentry that was not acquired + by doing a lookup in the parent directory. This includes "/", + "." and "..", as well as procfs-style symlinks and mountpoint + traversal. + + In this case, we are less concerned with whether the dentry is + still fully correct, but rather that the inode is still valid. + As with d_revalidate, most local filesystems will set this to + NULL since their dcache entries are always valid. + + This function has the same return code semantics as + d_revalidate. + + d_weak_revalidate is only called after leaving rcu-walk mode. + +``d_hash`` + called when the VFS adds a dentry to the hash table. The first + dentry passed to d_hash is the parent directory that the name is + to be hashed into. + + Same locking and synchronisation rules as d_compare regarding + what is safe to dereference etc. + +``d_compare`` + called to compare a dentry name with a given name. The first + dentry is the parent of the dentry to be compared, the second is + the child dentry. len and name string are properties of the + dentry to be compared. qstr is the name to compare it with. + + Must be constant and idempotent, and should not take locks if + possible, and should not or store into the dentry. Should not + dereference pointers outside the dentry without lots of care + (eg. d_parent, d_inode, d_name should not be used). + + However, our vfsmount is pinned, and RCU held, so the dentries + and inodes won't disappear, neither will our sb or filesystem + module. ->d_sb may be used. + + It is a tricky calling convention because it needs to be called + under "rcu-walk", ie. without any locks or references on things. + +``d_delete`` + called when the last reference to a dentry is dropped and the + dcache is deciding whether or not to cache it. Return 1 to + delete immediately, or 0 to cache the dentry. Default is NULL + which means to always cache a reachable dentry. d_delete must + be constant and idempotent. + +``d_init`` + called when a dentry is allocated + +``d_release`` + called when a dentry is really deallocated + +``d_iput`` + called when a dentry loses its inode (just prior to its being + deallocated). The default when this is NULL is that the VFS + calls iput(). If you define this method, you must call iput() + yourself + +``d_dname`` + called when the pathname of a dentry should be generated. + Useful for some pseudo filesystems (sockfs, pipefs, ...) to + delay pathname generation. (Instead of doing it when dentry is + created, it's done only when the path is needed.). Real + filesystems probably dont want to use it, because their dentries + are present in global dcache hash, so their hash should be an + invariant. As no lock is held, d_dname() should not try to + modify the dentry itself, unless appropriate SMP safety is used. + CAUTION : d_path() logic is quite tricky. The correct way to + return for example "Hello" is to put it at the end of the + buffer, and returns a pointer to the first char. + dynamic_dname() helper function is provided to take care of + this. + + Example : + +.. code-block:: c + + static char *pipefs_dname(struct dentry *dent, char *buffer, int buflen) + { + return dynamic_dname(dentry, buffer, buflen, "pipe:[%lu]", + dentry->d_inode->i_ino); + } + +``d_automount`` + called when an automount dentry is to be traversed (optional). + This should create a new VFS mount record and return the record + to the caller. The caller is supplied with a path parameter + giving the automount directory to describe the automount target + and the parent VFS mount record to provide inheritable mount + parameters. NULL should be returned if someone else managed to + make the automount first. If the vfsmount creation failed, then + an error code should be returned. If -EISDIR is returned, then + the directory will be treated as an ordinary directory and + returned to pathwalk to continue walking. + + If a vfsmount is returned, the caller will attempt to mount it + on the mountpoint and will remove the vfsmount from its + expiration list in the case of failure. The vfsmount should be + returned with 2 refs on it to prevent automatic expiration - the + caller will clean up the additional ref. + + This function is only used if DCACHE_NEED_AUTOMOUNT is set on + the dentry. This is set by __d_instantiate() if S_AUTOMOUNT is + set on the inode being added. + +``d_manage`` + called to allow the filesystem to manage the transition from a + dentry (optional). This allows autofs, for example, to hold up + clients waiting to explore behind a 'mountpoint' while letting + the daemon go past and construct the subtree there. 0 should be + returned to let the calling process continue. -EISDIR can be + returned to tell pathwalk to use this directory as an ordinary + directory and to ignore anything mounted on it and not to check + the automount flag. Any other error code will abort pathwalk + completely. + + If the 'rcu_walk' parameter is true, then the caller is doing a + pathwalk in RCU-walk mode. Sleeping is not permitted in this + mode, and the caller can be asked to leave it and call again by + returning -ECHILD. -EISDIR may also be returned to tell + pathwalk to ignore d_automount or any mounts. + + This function is only used if DCACHE_MANAGE_TRANSIT is set on + the dentry being transited from. + +``d_real`` + overlay/union type filesystems implement this method to return + one of the underlying dentries hidden by the overlay. It is + used in two different modes: + + Called from file_dentry() it returns the real dentry matching + the inode argument. The real dentry may be from a lower layer + already copied up, but still referenced from the file. This + mode is selected with a non-NULL inode argument. + + With NULL inode the topmost real underlying dentry is returned. + +Each dentry has a pointer to its parent dentry, as well as a hash list +of child dentries. Child dentries are basically like files in a +directory. + + +Directory Entry Cache API +-------------------------- + +There are a number of functions defined which permit a filesystem to +manipulate dentries: + +``dget`` + open a new handle for an existing dentry (this just increments + the usage count) + +``dput`` + close a handle for a dentry (decrements the usage count). If + the usage count drops to 0, and the dentry is still in its + parent's hash, the "d_delete" method is called to check whether + it should be cached. If it should not be cached, or if the + dentry is not hashed, it is deleted. Otherwise cached dentries + are put into an LRU list to be reclaimed on memory shortage. + +``d_drop`` + this unhashes a dentry from its parents hash list. A subsequent + call to dput() will deallocate the dentry if its usage count + drops to 0 + +``d_delete`` + delete a dentry. If there are no other open references to the + dentry then the dentry is turned into a negative dentry (the + d_iput() method is called). If there are other references, then + d_drop() is called instead + +``d_add`` + add a dentry to its parents hash list and then calls + d_instantiate() + +``d_instantiate`` + add a dentry to the alias hash list for the inode and updates + the "d_inode" member. The "i_count" member in the inode + structure should be set/incremented. If the inode pointer is + NULL, the dentry is called a "negative dentry". This function + is commonly called when an inode is created for an existing + negative dentry + +``d_lookup`` + look up a dentry given its parent and path name component It + looks up the child of that given name from the dcache hash + table. If it is found, the reference count is incremented and + the dentry is returned. The caller must use dput() to free the + dentry when it finishes using it. + + +Mount Options +============= + + +Parsing options +--------------- + +On mount and remount the filesystem is passed a string containing a +comma separated list of mount options. The options can have either of +these forms: + + option + option=value + +The <linux/parser.h> header defines an API that helps parse these +options. There are plenty of examples on how to use it in existing +filesystems. + + +Showing options +--------------- + +If a filesystem accepts mount options, it must define show_options() to +show all the currently active options. The rules are: + + - options MUST be shown which are not default or their values differ + from the default + + - options MAY be shown which are enabled by default or have their + default value + +Options used only internally between a mount helper and the kernel (such +as file descriptors), or which only have an effect during the mounting +(such as ones controlling the creation of a journal) are exempt from the +above rules. + +The underlying reason for the above rules is to make sure, that a mount +can be accurately replicated (e.g. umounting and mounting again) based +on the information found in /proc/mounts. + + +Resources +========= + +(Note some of these resources are not up-to-date with the latest kernel + version.) + +Creating Linux virtual filesystems. 2002 + <http://lwn.net/Articles/13325/> + +The Linux Virtual File-system Layer by Neil Brown. 1999 + <http://www.cse.unsw.edu.au/~neilb/oss/linux-commentary/vfs.html> + +A tour of the Linux VFS by Michael K. Johnson. 1996 + <http://www.tldp.org/LDP/khg/HyperNews/get/fs/vfstour.html> + +A small trail through the Linux kernel by Andries Brouwer. 2001 + <http://www.win.tue.nl/~aeb/linux/vfs/trail.html> diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt deleted file mode 100644 index 57fc576b1f3e..000000000000 --- a/Documentation/filesystems/vfs.txt +++ /dev/null @@ -1,1268 +0,0 @@ - - Overview of the Linux Virtual File System - - Original author: Richard Gooch <rgooch@atnf.csiro.au> - - Copyright (C) 1999 Richard Gooch - Copyright (C) 2005 Pekka Enberg - - This file is released under the GPLv2. - - -Introduction -============ - -The Virtual File System (also known as the Virtual Filesystem Switch) -is the software layer in the kernel that provides the filesystem -interface to userspace programs. It also provides an abstraction -within the kernel which allows different filesystem implementations to -coexist. - -VFS system calls open(2), stat(2), read(2), write(2), chmod(2) and so -on are called from a process context. Filesystem locking is described -in the document Documentation/filesystems/Locking. - - -Directory Entry Cache (dcache) ------------------------------- - -The VFS implements the open(2), stat(2), chmod(2), and similar system -calls. The pathname argument that is passed to them is used by the VFS -to search through the directory entry cache (also known as the dentry -cache or dcache). This provides a very fast look-up mechanism to -translate a pathname (filename) into a specific dentry. Dentries live -in RAM and are never saved to disc: they exist only for performance. - -The dentry cache is meant to be a view into your entire filespace. As -most computers cannot fit all dentries in the RAM at the same time, -some bits of the cache are missing. In order to resolve your pathname -into a dentry, the VFS may have to resort to creating dentries along -the way, and then loading the inode. This is done by looking up the -inode. - - -The Inode Object ----------------- - -An individual dentry usually has a pointer to an inode. Inodes are -filesystem objects such as regular files, directories, FIFOs and other -beasts. They live either on the disc (for block device filesystems) -or in the memory (for pseudo filesystems). Inodes that live on the -disc are copied into the memory when required and changes to the inode -are written back to disc. A single inode can be pointed to by multiple -dentries (hard links, for example, do this). - -To look up an inode requires that the VFS calls the lookup() method of -the parent directory inode. This method is installed by the specific -filesystem implementation that the inode lives in. Once the VFS has -the required dentry (and hence the inode), we can do all those boring -things like open(2) the file, or stat(2) it to peek at the inode -data. The stat(2) operation is fairly simple: once the VFS has the -dentry, it peeks at the inode data and passes some of it back to -userspace. - - -The File Object ---------------- - -Opening a file requires another operation: allocation of a file -structure (this is the kernel-side implementation of file -descriptors). The freshly allocated file structure is initialized with -a pointer to the dentry and a set of file operation member functions. -These are taken from the inode data. The open() file method is then -called so the specific filesystem implementation can do its work. You -can see that this is another switch performed by the VFS. The file -structure is placed into the file descriptor table for the process. - -Reading, writing and closing files (and other assorted VFS operations) -is done by using the userspace file descriptor to grab the appropriate -file structure, and then calling the required file structure method to -do whatever is required. For as long as the file is open, it keeps the -dentry in use, which in turn means that the VFS inode is still in use. - - -Registering and Mounting a Filesystem -===================================== - -To register and unregister a filesystem, use the following API -functions: - - #include <linux/fs.h> - - extern int register_filesystem(struct file_system_type *); - extern int unregister_filesystem(struct file_system_type *); - -The passed struct file_system_type describes your filesystem. When a -request is made to mount a filesystem onto a directory in your namespace, -the VFS will call the appropriate mount() method for the specific -filesystem. New vfsmount referring to the tree returned by ->mount() -will be attached to the mountpoint, so that when pathname resolution -reaches the mountpoint it will jump into the root of that vfsmount. - -You can see all filesystems that are registered to the kernel in the -file /proc/filesystems. - - -struct file_system_type ------------------------ - -This describes the filesystem. As of kernel 2.6.39, the following -members are defined: - -struct file_system_type { - const char *name; - int fs_flags; - struct dentry *(*mount) (struct file_system_type *, int, - const char *, void *); - void (*kill_sb) (struct super_block *); - struct module *owner; - struct file_system_type * next; - struct list_head fs_supers; - struct lock_class_key s_lock_key; - struct lock_class_key s_umount_key; -}; - - name: the name of the filesystem type, such as "ext2", "iso9660", - "msdos" and so on - - fs_flags: various flags (i.e. FS_REQUIRES_DEV, FS_NO_DCACHE, etc.) - - mount: the method to call when a new instance of this - filesystem should be mounted - - kill_sb: the method to call when an instance of this filesystem - should be shut down - - owner: for internal VFS use: you should initialize this to THIS_MODULE in - most cases. - - next: for internal VFS use: you should initialize this to NULL - - s_lock_key, s_umount_key: lockdep-specific - -The mount() method has the following arguments: - - struct file_system_type *fs_type: describes the filesystem, partly initialized - by the specific filesystem code - - int flags: mount flags - - const char *dev_name: the device name we are mounting. - - void *data: arbitrary mount options, usually comes as an ASCII - string (see "Mount Options" section) - -The mount() method must return the root dentry of the tree requested by -caller. An active reference to its superblock must be grabbed and the -superblock must be locked. On failure it should return ERR_PTR(error). - -The arguments match those of mount(2) and their interpretation -depends on filesystem type. E.g. for block filesystems, dev_name is -interpreted as block device name, that device is opened and if it -contains a suitable filesystem image the method creates and initializes -struct super_block accordingly, returning its root dentry to caller. - -->mount() may choose to return a subtree of existing filesystem - it -doesn't have to create a new one. The main result from the caller's -point of view is a reference to dentry at the root of (sub)tree to -be attached; creation of new superblock is a common side effect. - -The most interesting member of the superblock structure that the -mount() method fills in is the "s_op" field. This is a pointer to -a "struct super_operations" which describes the next level of the -filesystem implementation. - -Usually, a filesystem uses one of the generic mount() implementations -and provides a fill_super() callback instead. The generic variants are: - - mount_bdev: mount a filesystem residing on a block device - - mount_nodev: mount a filesystem that is not backed by a device - - mount_single: mount a filesystem which shares the instance between - all mounts - -A fill_super() callback implementation has the following arguments: - - struct super_block *sb: the superblock structure. The callback - must initialize this properly. - - void *data: arbitrary mount options, usually comes as an ASCII - string (see "Mount Options" section) - - int silent: whether or not to be silent on error - - -The Superblock Object -===================== - -A superblock object represents a mounted filesystem. - - -struct super_operations ------------------------ - -This describes how the VFS can manipulate the superblock of your -filesystem. As of kernel 2.6.22, the following members are defined: - -struct super_operations { - struct inode *(*alloc_inode)(struct super_block *sb); - void (*destroy_inode)(struct inode *); - - void (*dirty_inode) (struct inode *, int flags); - int (*write_inode) (struct inode *, int); - void (*drop_inode) (struct inode *); - void (*delete_inode) (struct inode *); - void (*put_super) (struct super_block *); - int (*sync_fs)(struct super_block *sb, int wait); - int (*freeze_fs) (struct super_block *); - int (*unfreeze_fs) (struct super_block *); - int (*statfs) (struct dentry *, struct kstatfs *); - int (*remount_fs) (struct super_block *, int *, char *); - void (*clear_inode) (struct inode *); - void (*umount_begin) (struct super_block *); - - int (*show_options)(struct seq_file *, struct dentry *); - - ssize_t (*quota_read)(struct super_block *, int, char *, size_t, loff_t); - ssize_t (*quota_write)(struct super_block *, int, const char *, size_t, loff_t); - int (*nr_cached_objects)(struct super_block *); - void (*free_cached_objects)(struct super_block *, int); -}; - -All methods are called without any locks being held, unless otherwise -noted. This means that most methods can block safely. All methods are -only called from a process context (i.e. not from an interrupt handler -or bottom half). - - alloc_inode: this method is called by alloc_inode() to allocate memory - for struct inode and initialize it. If this function is not - defined, a simple 'struct inode' is allocated. Normally - alloc_inode will be used to allocate a larger structure which - contains a 'struct inode' embedded within it. - - destroy_inode: this method is called by destroy_inode() to release - resources allocated for struct inode. It is only required if - ->alloc_inode was defined and simply undoes anything done by - ->alloc_inode. - - dirty_inode: this method is called by the VFS to mark an inode dirty. - - write_inode: this method is called when the VFS needs to write an - inode to disc. The second parameter indicates whether the write - should be synchronous or not, not all filesystems check this flag. - - drop_inode: called when the last access to the inode is dropped, - with the inode->i_lock spinlock held. - - This method should be either NULL (normal UNIX filesystem - semantics) or "generic_delete_inode" (for filesystems that do not - want to cache inodes - causing "delete_inode" to always be - called regardless of the value of i_nlink) - - The "generic_delete_inode()" behavior is equivalent to the - old practice of using "force_delete" in the put_inode() case, - but does not have the races that the "force_delete()" approach - had. - - delete_inode: called when the VFS wants to delete an inode - - put_super: called when the VFS wishes to free the superblock - (i.e. unmount). This is called with the superblock lock held - - sync_fs: called when VFS is writing out all dirty data associated with - a superblock. The second parameter indicates whether the method - should wait until the write out has been completed. Optional. - - freeze_fs: called when VFS is locking a filesystem and - forcing it into a consistent state. This method is currently - used by the Logical Volume Manager (LVM). - - unfreeze_fs: called when VFS is unlocking a filesystem and making it writable - again. - - statfs: called when the VFS needs to get filesystem statistics. - - remount_fs: called when the filesystem is remounted. This is called - with the kernel lock held - - clear_inode: called then the VFS clears the inode. Optional - - umount_begin: called when the VFS is unmounting a filesystem. - - show_options: called by the VFS to show mount options for - /proc/<pid>/mounts. (see "Mount Options" section) - - quota_read: called by the VFS to read from filesystem quota file. - - quota_write: called by the VFS to write to filesystem quota file. - - nr_cached_objects: called by the sb cache shrinking function for the - filesystem to return the number of freeable cached objects it contains. - Optional. - - free_cache_objects: called by the sb cache shrinking function for the - filesystem to scan the number of objects indicated to try to free them. - Optional, but any filesystem implementing this method needs to also - implement ->nr_cached_objects for it to be called correctly. - - We can't do anything with any errors that the filesystem might - encountered, hence the void return type. This will never be called if - the VM is trying to reclaim under GFP_NOFS conditions, hence this - method does not need to handle that situation itself. - - Implementations must include conditional reschedule calls inside any - scanning loop that is done. This allows the VFS to determine - appropriate scan batch sizes without having to worry about whether - implementations will cause holdoff problems due to large scan batch - sizes. - -Whoever sets up the inode is responsible for filling in the "i_op" field. This -is a pointer to a "struct inode_operations" which describes the methods that -can be performed on individual inodes. - -struct xattr_handlers ---------------------- - -On filesystems that support extended attributes (xattrs), the s_xattr -superblock field points to a NULL-terminated array of xattr handlers. Extended -attributes are name:value pairs. - - name: Indicates that the handler matches attributes with the specified name - (such as "system.posix_acl_access"); the prefix field must be NULL. - - prefix: Indicates that the handler matches all attributes with the specified - name prefix (such as "user."); the name field must be NULL. - - list: Determine if attributes matching this xattr handler should be listed - for a particular dentry. Used by some listxattr implementations like - generic_listxattr. - - get: Called by the VFS to get the value of a particular extended attribute. - This method is called by the getxattr(2) system call. - - set: Called by the VFS to set the value of a particular extended attribute. - When the new value is NULL, called to remove a particular extended - attribute. This method is called by the the setxattr(2) and - removexattr(2) system calls. - -When none of the xattr handlers of a filesystem match the specified attribute -name or when a filesystem doesn't support extended attributes, the various -*xattr(2) system calls return -EOPNOTSUPP. - - -The Inode Object -================ - -An inode object represents an object within the filesystem. - - -struct inode_operations ------------------------ - -This describes how the VFS can manipulate an inode in your -filesystem. As of kernel 2.6.22, the following members are defined: - -struct inode_operations { - int (*create) (struct inode *,struct dentry *, umode_t, bool); - struct dentry * (*lookup) (struct inode *,struct dentry *, unsigned int); - int (*link) (struct dentry *,struct inode *,struct dentry *); - int (*unlink) (struct inode *,struct dentry *); - int (*symlink) (struct inode *,struct dentry *,const char *); - int (*mkdir) (struct inode *,struct dentry *,umode_t); - int (*rmdir) (struct inode *,struct dentry *); - int (*mknod) (struct inode *,struct dentry *,umode_t,dev_t); - int (*rename) (struct inode *, struct dentry *, - struct inode *, struct dentry *, unsigned int); - int (*readlink) (struct dentry *, char __user *,int); - const char *(*get_link) (struct dentry *, struct inode *, - struct delayed_call *); - int (*permission) (struct inode *, int); - int (*get_acl)(struct inode *, int); - int (*setattr) (struct dentry *, struct iattr *); - int (*getattr) (const struct path *, struct kstat *, u32, unsigned int); - ssize_t (*listxattr) (struct dentry *, char *, size_t); - void (*update_time)(struct inode *, struct timespec *, int); - int (*atomic_open)(struct inode *, struct dentry *, struct file *, - unsigned open_flag, umode_t create_mode); - int (*tmpfile) (struct inode *, struct dentry *, umode_t); -}; - -Again, all methods are called without any locks being held, unless -otherwise noted. - - create: called by the open(2) and creat(2) system calls. Only - required if you want to support regular files. The dentry you - get should not have an inode (i.e. it should be a negative - dentry). Here you will probably call d_instantiate() with the - dentry and the newly created inode - - lookup: called when the VFS needs to look up an inode in a parent - directory. The name to look for is found in the dentry. This - method must call d_add() to insert the found inode into the - dentry. The "i_count" field in the inode structure should be - incremented. If the named inode does not exist a NULL inode - should be inserted into the dentry (this is called a negative - dentry). Returning an error code from this routine must only - be done on a real error, otherwise creating inodes with system - calls like create(2), mknod(2), mkdir(2) and so on will fail. - If you wish to overload the dentry methods then you should - initialise the "d_dop" field in the dentry; this is a pointer - to a struct "dentry_operations". - This method is called with the directory inode semaphore held - - link: called by the link(2) system call. Only required if you want - to support hard links. You will probably need to call - d_instantiate() just as you would in the create() method - - unlink: called by the unlink(2) system call. Only required if you - want to support deleting inodes - - symlink: called by the symlink(2) system call. Only required if you - want to support symlinks. You will probably need to call - d_instantiate() just as you would in the create() method - - mkdir: called by the mkdir(2) system call. Only required if you want - to support creating subdirectories. You will probably need to - call d_instantiate() just as you would in the create() method - - rmdir: called by the rmdir(2) system call. Only required if you want - to support deleting subdirectories - - mknod: called by the mknod(2) system call to create a device (char, - block) inode or a named pipe (FIFO) or socket. Only required - if you want to support creating these types of inodes. You - will probably need to call d_instantiate() just as you would - in the create() method - - rename: called by the rename(2) system call to rename the object to - have the parent and name given by the second inode and dentry. - - The filesystem must return -EINVAL for any unsupported or - unknown flags. Currently the following flags are implemented: - (1) RENAME_NOREPLACE: this flag indicates that if the target - of the rename exists the rename should fail with -EEXIST - instead of replacing the target. The VFS already checks for - existence, so for local filesystems the RENAME_NOREPLACE - implementation is equivalent to plain rename. - (2) RENAME_EXCHANGE: exchange source and target. Both must - exist; this is checked by the VFS. Unlike plain rename, - source and target may be of different type. - - get_link: called by the VFS to follow a symbolic link to the - inode it points to. Only required if you want to support - symbolic links. This method returns the symlink body - to traverse (and possibly resets the current position with - nd_jump_link()). If the body won't go away until the inode - is gone, nothing else is needed; if it needs to be otherwise - pinned, arrange for its release by having get_link(..., ..., done) - do set_delayed_call(done, destructor, argument). - In that case destructor(argument) will be called once VFS is - done with the body you've returned. - May be called in RCU mode; that is indicated by NULL dentry - argument. If request can't be handled without leaving RCU mode, - have it return ERR_PTR(-ECHILD). - - If the filesystem stores the symlink target in ->i_link, the - VFS may use it directly without calling ->get_link(); however, - ->get_link() must still be provided. ->i_link must not be - freed until after an RCU grace period. Writing to ->i_link - post-iget() time requires a 'release' memory barrier. - - readlink: this is now just an override for use by readlink(2) for the - cases when ->get_link uses nd_jump_link() or object is not in - fact a symlink. Normally filesystems should only implement - ->get_link for symlinks and readlink(2) will automatically use - that. - - permission: called by the VFS to check for access rights on a POSIX-like - filesystem. - - May be called in rcu-walk mode (mask & MAY_NOT_BLOCK). If in rcu-walk - mode, the filesystem must check the permission without blocking or - storing to the inode. - - If a situation is encountered that rcu-walk cannot handle, return - -ECHILD and it will be called again in ref-walk mode. - - setattr: called by the VFS to set attributes for a file. This method - is called by chmod(2) and related system calls. - - getattr: called by the VFS to get attributes of a file. This method - is called by stat(2) and related system calls. - - listxattr: called by the VFS to list all extended attributes for a - given file. This method is called by the listxattr(2) system call. - - update_time: called by the VFS to update a specific time or the i_version of - an inode. If this is not defined the VFS will update the inode itself - and call mark_inode_dirty_sync. - - atomic_open: called on the last component of an open. Using this optional - method the filesystem can look up, possibly create and open the file in - one atomic operation. If it wants to leave actual opening to the - caller (e.g. if the file turned out to be a symlink, device, or just - something filesystem won't do atomic open for), it may signal this by - returning finish_no_open(file, dentry). This method is only called if - the last component is negative or needs lookup. Cached positive dentries - are still handled by f_op->open(). If the file was created, - FMODE_CREATED flag should be set in file->f_mode. In case of O_EXCL - the method must only succeed if the file didn't exist and hence FMODE_CREATED - shall always be set on success. - - tmpfile: called in the end of O_TMPFILE open(). Optional, equivalent to - atomically creating, opening and unlinking a file in given directory. - -The Address Space Object -======================== - -The address space object is used to group and manage pages in the page -cache. It can be used to keep track of the pages in a file (or -anything else) and also track the mapping of sections of the file into -process address spaces. - -There are a number of distinct yet related services that an -address-space can provide. These include communicating memory -pressure, page lookup by address, and keeping track of pages tagged as -Dirty or Writeback. - -The first can be used independently to the others. The VM can try to -either write dirty pages in order to clean them, or release clean -pages in order to reuse them. To do this it can call the ->writepage -method on dirty pages, and ->releasepage on clean pages with -PagePrivate set. Clean pages without PagePrivate and with no external -references will be released without notice being given to the -address_space. - -To achieve this functionality, pages need to be placed on an LRU with -lru_cache_add and mark_page_active needs to be called whenever the -page is used. - -Pages are normally kept in a radix tree index by ->index. This tree -maintains information about the PG_Dirty and PG_Writeback status of -each page, so that pages with either of these flags can be found -quickly. - -The Dirty tag is primarily used by mpage_writepages - the default -->writepages method. It uses the tag to find dirty pages to call -->writepage on. If mpage_writepages is not used (i.e. the address -provides its own ->writepages) , the PAGECACHE_TAG_DIRTY tag is -almost unused. write_inode_now and sync_inode do use it (through -__sync_single_inode) to check if ->writepages has been successful in -writing out the whole address_space. - -The Writeback tag is used by filemap*wait* and sync_page* functions, -via filemap_fdatawait_range, to wait for all writeback to complete. - -An address_space handler may attach extra information to a page, -typically using the 'private' field in the 'struct page'. If such -information is attached, the PG_Private flag should be set. This will -cause various VM routines to make extra calls into the address_space -handler to deal with that data. - -An address space acts as an intermediate between storage and -application. Data is read into the address space a whole page at a -time, and provided to the application either by copying of the page, -or by memory-mapping the page. -Data is written into the address space by the application, and then -written-back to storage typically in whole pages, however the -address_space has finer control of write sizes. - -The read process essentially only requires 'readpage'. The write -process is more complicated and uses write_begin/write_end or -set_page_dirty to write data into the address_space, and writepage -and writepages to writeback data to storage. - -Adding and removing pages to/from an address_space is protected by the -inode's i_mutex. - -When data is written to a page, the PG_Dirty flag should be set. It -typically remains set until writepage asks for it to be written. This -should clear PG_Dirty and set PG_Writeback. It can be actually -written at any point after PG_Dirty is clear. Once it is known to be -safe, PG_Writeback is cleared. - -Writeback makes use of a writeback_control structure to direct the -operations. This gives the the writepage and writepages operations some -information about the nature of and reason for the writeback request, -and the constraints under which it is being done. It is also used to -return information back to the caller about the result of a writepage or -writepages request. - -Handling errors during writeback --------------------------------- -Most applications that do buffered I/O will periodically call a file -synchronization call (fsync, fdatasync, msync or sync_file_range) to -ensure that data written has made it to the backing store. When there -is an error during writeback, they expect that error to be reported when -a file sync request is made. After an error has been reported on one -request, subsequent requests on the same file descriptor should return -0, unless further writeback errors have occurred since the previous file -syncronization. - -Ideally, the kernel would report errors only on file descriptions on -which writes were done that subsequently failed to be written back. The -generic pagecache infrastructure does not track the file descriptions -that have dirtied each individual page however, so determining which -file descriptors should get back an error is not possible. - -Instead, the generic writeback error tracking infrastructure in the -kernel settles for reporting errors to fsync on all file descriptions -that were open at the time that the error occurred. In a situation with -multiple writers, all of them will get back an error on a subsequent fsync, -even if all of the writes done through that particular file descriptor -succeeded (or even if there were no writes on that file descriptor at all). - -Filesystems that wish to use this infrastructure should call -mapping_set_error to record the error in the address_space when it -occurs. Then, after writing back data from the pagecache in their -file->fsync operation, they should call file_check_and_advance_wb_err to -ensure that the struct file's error cursor has advanced to the correct -point in the stream of errors emitted by the backing device(s). - -struct address_space_operations -------------------------------- - -This describes how the VFS can manipulate mapping of a file to page cache in -your filesystem. The following members are defined: - -struct address_space_operations { - int (*writepage)(struct page *page, struct writeback_control *wbc); - int (*readpage)(struct file *, struct page *); - int (*writepages)(struct address_space *, struct writeback_control *); - int (*set_page_dirty)(struct page *page); - int (*readpages)(struct file *filp, struct address_space *mapping, - struct list_head *pages, unsigned nr_pages); - int (*write_begin)(struct file *, struct address_space *mapping, - loff_t pos, unsigned len, unsigned flags, - struct page **pagep, void **fsdata); - int (*write_end)(struct file *, struct address_space *mapping, - loff_t pos, unsigned len, unsigned copied, - struct page *page, void *fsdata); - sector_t (*bmap)(struct address_space *, sector_t); - void (*invalidatepage) (struct page *, unsigned int, unsigned int); - int (*releasepage) (struct page *, int); - void (*freepage)(struct page *); - ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter); - /* isolate a page for migration */ - bool (*isolate_page) (struct page *, isolate_mode_t); - /* migrate the contents of a page to the specified target */ - int (*migratepage) (struct page *, struct page *); - /* put migration-failed page back to right list */ - void (*putback_page) (struct page *); - int (*launder_page) (struct page *); - - int (*is_partially_uptodate) (struct page *, unsigned long, - unsigned long); - void (*is_dirty_writeback) (struct page *, bool *, bool *); - int (*error_remove_page) (struct mapping *mapping, struct page *page); - int (*swap_activate)(struct file *); - int (*swap_deactivate)(struct file *); -}; - - writepage: called by the VM to write a dirty page to backing store. - This may happen for data integrity reasons (i.e. 'sync'), or - to free up memory (flush). The difference can be seen in - wbc->sync_mode. - The PG_Dirty flag has been cleared and PageLocked is true. - writepage should start writeout, should set PG_Writeback, - and should make sure the page is unlocked, either synchronously - or asynchronously when the write operation completes. - - If wbc->sync_mode is WB_SYNC_NONE, ->writepage doesn't have to - try too hard if there are problems, and may choose to write out - other pages from the mapping if that is easier (e.g. due to - internal dependencies). If it chooses not to start writeout, it - should return AOP_WRITEPAGE_ACTIVATE so that the VM will not keep - calling ->writepage on that page. - - See the file "Locking" for more details. - - readpage: called by the VM to read a page from backing store. - The page will be Locked when readpage is called, and should be - unlocked and marked uptodate once the read completes. - If ->readpage discovers that it needs to unlock the page for - some reason, it can do so, and then return AOP_TRUNCATED_PAGE. - In this case, the page will be relocated, relocked and if - that all succeeds, ->readpage will be called again. - - writepages: called by the VM to write out pages associated with the - address_space object. If wbc->sync_mode is WBC_SYNC_ALL, then - the writeback_control will specify a range of pages that must be - written out. If it is WBC_SYNC_NONE, then a nr_to_write is given - and that many pages should be written if possible. - If no ->writepages is given, then mpage_writepages is used - instead. This will choose pages from the address space that are - tagged as DIRTY and will pass them to ->writepage. - - set_page_dirty: called by the VM to set a page dirty. - This is particularly needed if an address space attaches - private data to a page, and that data needs to be updated when - a page is dirtied. This is called, for example, when a memory - mapped page gets modified. - If defined, it should set the PageDirty flag, and the - PAGECACHE_TAG_DIRTY tag in the radix tree. - - readpages: called by the VM to read pages associated with the address_space - object. This is essentially just a vector version of - readpage. Instead of just one page, several pages are - requested. - readpages is only used for read-ahead, so read errors are - ignored. If anything goes wrong, feel free to give up. - - write_begin: - Called by the generic buffered write code to ask the filesystem to - prepare to write len bytes at the given offset in the file. The - address_space should check that the write will be able to complete, - by allocating space if necessary and doing any other internal - housekeeping. If the write will update parts of any basic-blocks on - storage, then those blocks should be pre-read (if they haven't been - read already) so that the updated blocks can be written out properly. - - The filesystem must return the locked pagecache page for the specified - offset, in *pagep, for the caller to write into. - - It must be able to cope with short writes (where the length passed to - write_begin is greater than the number of bytes copied into the page). - - flags is a field for AOP_FLAG_xxx flags, described in - include/linux/fs.h. - - A void * may be returned in fsdata, which then gets passed into - write_end. - - Returns 0 on success; < 0 on failure (which is the error code), in - which case write_end is not called. - - write_end: After a successful write_begin, and data copy, write_end must - be called. len is the original len passed to write_begin, and copied - is the amount that was able to be copied. - - The filesystem must take care of unlocking the page and releasing it - refcount, and updating i_size. - - Returns < 0 on failure, otherwise the number of bytes (<= 'copied') - that were able to be copied into pagecache. - - bmap: called by the VFS to map a logical block offset within object to - physical block number. This method is used by the FIBMAP - ioctl and for working with swap-files. To be able to swap to - a file, the file must have a stable mapping to a block - device. The swap system does not go through the filesystem - but instead uses bmap to find out where the blocks in the file - are and uses those addresses directly. - - invalidatepage: If a page has PagePrivate set, then invalidatepage - will be called when part or all of the page is to be removed - from the address space. This generally corresponds to either a - truncation, punch hole or a complete invalidation of the address - space (in the latter case 'offset' will always be 0 and 'length' - will be PAGE_SIZE). Any private data associated with the page - should be updated to reflect this truncation. If offset is 0 and - length is PAGE_SIZE, then the private data should be released, - because the page must be able to be completely discarded. This may - be done by calling the ->releasepage function, but in this case the - release MUST succeed. - - releasepage: releasepage is called on PagePrivate pages to indicate - that the page should be freed if possible. ->releasepage - should remove any private data from the page and clear the - PagePrivate flag. If releasepage() fails for some reason, it must - indicate failure with a 0 return value. - releasepage() is used in two distinct though related cases. The - first is when the VM finds a clean page with no active users and - wants to make it a free page. If ->releasepage succeeds, the - page will be removed from the address_space and become free. - - The second case is when a request has been made to invalidate - some or all pages in an address_space. This can happen - through the fadvise(POSIX_FADV_DONTNEED) system call or by the - filesystem explicitly requesting it as nfs and 9fs do (when - they believe the cache may be out of date with storage) by - calling invalidate_inode_pages2(). - If the filesystem makes such a call, and needs to be certain - that all pages are invalidated, then its releasepage will - need to ensure this. Possibly it can clear the PageUptodate - bit if it cannot free private data yet. - - freepage: freepage is called once the page is no longer visible in - the page cache in order to allow the cleanup of any private - data. Since it may be called by the memory reclaimer, it - should not assume that the original address_space mapping still - exists, and it should not block. - - direct_IO: called by the generic read/write routines to perform - direct_IO - that is IO requests which bypass the page cache - and transfer data directly between the storage and the - application's address space. - - isolate_page: Called by the VM when isolating a movable non-lru page. - If page is successfully isolated, VM marks the page as PG_isolated - via __SetPageIsolated. - - migrate_page: This is used to compact the physical memory usage. - If the VM wants to relocate a page (maybe off a memory card - that is signalling imminent failure) it will pass a new page - and an old page to this function. migrate_page should - transfer any private data across and update any references - that it has to the page. - - putback_page: Called by the VM when isolated page's migration fails. - - launder_page: Called before freeing a page - it writes back the dirty page. To - prevent redirtying the page, it is kept locked during the whole - operation. - - is_partially_uptodate: Called by the VM when reading a file through the - pagecache when the underlying blocksize != pagesize. If the required - block is up to date then the read can complete without needing the IO - to bring the whole page up to date. - - is_dirty_writeback: Called by the VM when attempting to reclaim a page. - The VM uses dirty and writeback information to determine if it needs - to stall to allow flushers a chance to complete some IO. Ordinarily - it can use PageDirty and PageWriteback but some filesystems have - more complex state (unstable pages in NFS prevent reclaim) or - do not set those flags due to locking problems. This callback - allows a filesystem to indicate to the VM if a page should be - treated as dirty or writeback for the purposes of stalling. - - error_remove_page: normally set to generic_error_remove_page if truncation - is ok for this address space. Used for memory failure handling. - Setting this implies you deal with pages going away under you, - unless you have them locked or reference counts increased. - - swap_activate: Called when swapon is used on a file to allocate - space if necessary and pin the block lookup information in - memory. A return value of zero indicates success, - in which case this file can be used to back swapspace. - - swap_deactivate: Called during swapoff on files where swap_activate - was successful. - - -The File Object -=============== - -A file object represents a file opened by a process. This is also known -as an "open file description" in POSIX parlance. - - -struct file_operations ----------------------- - -This describes how the VFS can manipulate an open file. As of kernel -4.18, the following members are defined: - -struct file_operations { - struct module *owner; - loff_t (*llseek) (struct file *, loff_t, int); - ssize_t (*read) (struct file *, char __user *, size_t, loff_t *); - ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); - ssize_t (*read_iter) (struct kiocb *, struct iov_iter *); - ssize_t (*write_iter) (struct kiocb *, struct iov_iter *); - int (*iopoll)(struct kiocb *kiocb, bool spin); - int (*iterate) (struct file *, struct dir_context *); - int (*iterate_shared) (struct file *, struct dir_context *); - __poll_t (*poll) (struct file *, struct poll_table_struct *); - long (*unlocked_ioctl) (struct file *, unsigned int, unsigned long); - long (*compat_ioctl) (struct file *, unsigned int, unsigned long); - int (*mmap) (struct file *, struct vm_area_struct *); - int (*open) (struct inode *, struct file *); - int (*flush) (struct file *, fl_owner_t id); - int (*release) (struct inode *, struct file *); - int (*fsync) (struct file *, loff_t, loff_t, int datasync); - int (*fasync) (int, struct file *, int); - int (*lock) (struct file *, int, struct file_lock *); - ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int); - unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long); - int (*check_flags)(int); - int (*flock) (struct file *, int, struct file_lock *); - ssize_t (*splice_write)(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int); - ssize_t (*splice_read)(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); - int (*setlease)(struct file *, long, struct file_lock **, void **); - long (*fallocate)(struct file *file, int mode, loff_t offset, - loff_t len); - void (*show_fdinfo)(struct seq_file *m, struct file *f); -#ifndef CONFIG_MMU - unsigned (*mmap_capabilities)(struct file *); -#endif - ssize_t (*copy_file_range)(struct file *, loff_t, struct file *, loff_t, size_t, unsigned int); - loff_t (*remap_file_range)(struct file *file_in, loff_t pos_in, - struct file *file_out, loff_t pos_out, - loff_t len, unsigned int remap_flags); - int (*fadvise)(struct file *, loff_t, loff_t, int); -}; - -Again, all methods are called without any locks being held, unless -otherwise noted. - - llseek: called when the VFS needs to move the file position index - - read: called by read(2) and related system calls - - read_iter: possibly asynchronous read with iov_iter as destination - - write: called by write(2) and related system calls - - write_iter: possibly asynchronous write with iov_iter as source - - iopoll: called when aio wants to poll for completions on HIPRI iocbs - - iterate: called when the VFS needs to read the directory contents - - iterate_shared: called when the VFS needs to read the directory contents - when filesystem supports concurrent dir iterators - - poll: called by the VFS when a process wants to check if there is - activity on this file and (optionally) go to sleep until there - is activity. Called by the select(2) and poll(2) system calls - - unlocked_ioctl: called by the ioctl(2) system call. - - compat_ioctl: called by the ioctl(2) system call when 32 bit system calls - are used on 64 bit kernels. - - mmap: called by the mmap(2) system call - - open: called by the VFS when an inode should be opened. When the VFS - opens a file, it creates a new "struct file". It then calls the - open method for the newly allocated file structure. You might - think that the open method really belongs in - "struct inode_operations", and you may be right. I think it's - done the way it is because it makes filesystems simpler to - implement. The open() method is a good place to initialize the - "private_data" member in the file structure if you want to point - to a device structure - - flush: called by the close(2) system call to flush a file - - release: called when the last reference to an open file is closed - - fsync: called by the fsync(2) system call. Also see the section above - entitled "Handling errors during writeback". - - fasync: called by the fcntl(2) system call when asynchronous - (non-blocking) mode is enabled for a file - - lock: called by the fcntl(2) system call for F_GETLK, F_SETLK, and F_SETLKW - commands - - get_unmapped_area: called by the mmap(2) system call - - check_flags: called by the fcntl(2) system call for F_SETFL command - - flock: called by the flock(2) system call - - splice_write: called by the VFS to splice data from a pipe to a file. This - method is used by the splice(2) system call - - splice_read: called by the VFS to splice data from file to a pipe. This - method is used by the splice(2) system call - - setlease: called by the VFS to set or release a file lock lease. setlease - implementations should call generic_setlease to record or remove - the lease in the inode after setting it. - - fallocate: called by the VFS to preallocate blocks or punch a hole. - - copy_file_range: called by the copy_file_range(2) system call. - - remap_file_range: called by the ioctl(2) system call for FICLONERANGE and - FICLONE and FIDEDUPERANGE commands to remap file ranges. An - implementation should remap len bytes at pos_in of the source file into - the dest file at pos_out. Implementations must handle callers passing - in len == 0; this means "remap to the end of the source file". The - return value should the number of bytes remapped, or the usual - negative error code if errors occurred before any bytes were remapped. - The remap_flags parameter accepts REMAP_FILE_* flags. If - REMAP_FILE_DEDUP is set then the implementation must only remap if the - requested file ranges have identical contents. If REMAP_CAN_SHORTEN is - set, the caller is ok with the implementation shortening the request - length to satisfy alignment or EOF requirements (or any other reason). - - fadvise: possibly called by the fadvise64() system call. - -Note that the file operations are implemented by the specific -filesystem in which the inode resides. When opening a device node -(character or block special) most filesystems will call special -support routines in the VFS which will locate the required device -driver information. These support routines replace the filesystem file -operations with those for the device driver, and then proceed to call -the new open() method for the file. This is how opening a device file -in the filesystem eventually ends up calling the device driver open() -method. - - -Directory Entry Cache (dcache) -============================== - - -struct dentry_operations ------------------------- - -This describes how a filesystem can overload the standard dentry -operations. Dentries and the dcache are the domain of the VFS and the -individual filesystem implementations. Device drivers have no business -here. These methods may be set to NULL, as they are either optional or -the VFS uses a default. As of kernel 2.6.22, the following members are -defined: - -struct dentry_operations { - int (*d_revalidate)(struct dentry *, unsigned int); - int (*d_weak_revalidate)(struct dentry *, unsigned int); - int (*d_hash)(const struct dentry *, struct qstr *); - int (*d_compare)(const struct dentry *, - unsigned int, const char *, const struct qstr *); - int (*d_delete)(const struct dentry *); - int (*d_init)(struct dentry *); - void (*d_release)(struct dentry *); - void (*d_iput)(struct dentry *, struct inode *); - char *(*d_dname)(struct dentry *, char *, int); - struct vfsmount *(*d_automount)(struct path *); - int (*d_manage)(const struct path *, bool); - struct dentry *(*d_real)(struct dentry *, const struct inode *); -}; - - d_revalidate: called when the VFS needs to revalidate a dentry. This - is called whenever a name look-up finds a dentry in the - dcache. Most local filesystems leave this as NULL, because all their - dentries in the dcache are valid. Network filesystems are different - since things can change on the server without the client necessarily - being aware of it. - - This function should return a positive value if the dentry is still - valid, and zero or a negative error code if it isn't. - - d_revalidate may be called in rcu-walk mode (flags & LOOKUP_RCU). - If in rcu-walk mode, the filesystem must revalidate the dentry without - blocking or storing to the dentry, d_parent and d_inode should not be - used without care (because they can change and, in d_inode case, even - become NULL under us). - - If a situation is encountered that rcu-walk cannot handle, return - -ECHILD and it will be called again in ref-walk mode. - - d_weak_revalidate: called when the VFS needs to revalidate a "jumped" dentry. - This is called when a path-walk ends at dentry that was not acquired by - doing a lookup in the parent directory. This includes "/", "." and "..", - as well as procfs-style symlinks and mountpoint traversal. - - In this case, we are less concerned with whether the dentry is still - fully correct, but rather that the inode is still valid. As with - d_revalidate, most local filesystems will set this to NULL since their - dcache entries are always valid. - - This function has the same return code semantics as d_revalidate. - - d_weak_revalidate is only called after leaving rcu-walk mode. - - d_hash: called when the VFS adds a dentry to the hash table. The first - dentry passed to d_hash is the parent directory that the name is - to be hashed into. - - Same locking and synchronisation rules as d_compare regarding - what is safe to dereference etc. - - d_compare: called to compare a dentry name with a given name. The first - dentry is the parent of the dentry to be compared, the second is - the child dentry. len and name string are properties of the dentry - to be compared. qstr is the name to compare it with. - - Must be constant and idempotent, and should not take locks if - possible, and should not or store into the dentry. - Should not dereference pointers outside the dentry without - lots of care (eg. d_parent, d_inode, d_name should not be used). - - However, our vfsmount is pinned, and RCU held, so the dentries and - inodes won't disappear, neither will our sb or filesystem module. - ->d_sb may be used. - - It is a tricky calling convention because it needs to be called under - "rcu-walk", ie. without any locks or references on things. - - d_delete: called when the last reference to a dentry is dropped and the - dcache is deciding whether or not to cache it. Return 1 to delete - immediately, or 0 to cache the dentry. Default is NULL which means to - always cache a reachable dentry. d_delete must be constant and - idempotent. - - d_init: called when a dentry is allocated - - d_release: called when a dentry is really deallocated - - d_iput: called when a dentry loses its inode (just prior to its - being deallocated). The default when this is NULL is that the - VFS calls iput(). If you define this method, you must call - iput() yourself - - d_dname: called when the pathname of a dentry should be generated. - Useful for some pseudo filesystems (sockfs, pipefs, ...) to delay - pathname generation. (Instead of doing it when dentry is created, - it's done only when the path is needed.). Real filesystems probably - dont want to use it, because their dentries are present in global - dcache hash, so their hash should be an invariant. As no lock is - held, d_dname() should not try to modify the dentry itself, unless - appropriate SMP safety is used. CAUTION : d_path() logic is quite - tricky. The correct way to return for example "Hello" is to put it - at the end of the buffer, and returns a pointer to the first char. - dynamic_dname() helper function is provided to take care of this. - - Example : - - static char *pipefs_dname(struct dentry *dent, char *buffer, int buflen) - { - return dynamic_dname(dentry, buffer, buflen, "pipe:[%lu]", - dentry->d_inode->i_ino); - } - - d_automount: called when an automount dentry is to be traversed (optional). - This should create a new VFS mount record and return the record to the - caller. The caller is supplied with a path parameter giving the - automount directory to describe the automount target and the parent - VFS mount record to provide inheritable mount parameters. NULL should - be returned if someone else managed to make the automount first. If - the vfsmount creation failed, then an error code should be returned. - If -EISDIR is returned, then the directory will be treated as an - ordinary directory and returned to pathwalk to continue walking. - - If a vfsmount is returned, the caller will attempt to mount it on the - mountpoint and will remove the vfsmount from its expiration list in - the case of failure. The vfsmount should be returned with 2 refs on - it to prevent automatic expiration - the caller will clean up the - additional ref. - - This function is only used if DCACHE_NEED_AUTOMOUNT is set on the - dentry. This is set by __d_instantiate() if S_AUTOMOUNT is set on the - inode being added. - - d_manage: called to allow the filesystem to manage the transition from a - dentry (optional). This allows autofs, for example, to hold up clients - waiting to explore behind a 'mountpoint' while letting the daemon go - past and construct the subtree there. 0 should be returned to let the - calling process continue. -EISDIR can be returned to tell pathwalk to - use this directory as an ordinary directory and to ignore anything - mounted on it and not to check the automount flag. Any other error - code will abort pathwalk completely. - - If the 'rcu_walk' parameter is true, then the caller is doing a - pathwalk in RCU-walk mode. Sleeping is not permitted in this mode, - and the caller can be asked to leave it and call again by returning - -ECHILD. -EISDIR may also be returned to tell pathwalk to - ignore d_automount or any mounts. - - This function is only used if DCACHE_MANAGE_TRANSIT is set on the - dentry being transited from. - - d_real: overlay/union type filesystems implement this method to return one of - the underlying dentries hidden by the overlay. It is used in two - different modes: - - Called from file_dentry() it returns the real dentry matching the inode - argument. The real dentry may be from a lower layer already copied up, - but still referenced from the file. This mode is selected with a - non-NULL inode argument. - - With NULL inode the topmost real underlying dentry is returned. - -Each dentry has a pointer to its parent dentry, as well as a hash list -of child dentries. Child dentries are basically like files in a -directory. - - -Directory Entry Cache API --------------------------- - -There are a number of functions defined which permit a filesystem to -manipulate dentries: - - dget: open a new handle for an existing dentry (this just increments - the usage count) - - dput: close a handle for a dentry (decrements the usage count). If - the usage count drops to 0, and the dentry is still in its - parent's hash, the "d_delete" method is called to check whether - it should be cached. If it should not be cached, or if the dentry - is not hashed, it is deleted. Otherwise cached dentries are put - into an LRU list to be reclaimed on memory shortage. - - d_drop: this unhashes a dentry from its parents hash list. A - subsequent call to dput() will deallocate the dentry if its - usage count drops to 0 - - d_delete: delete a dentry. If there are no other open references to - the dentry then the dentry is turned into a negative dentry - (the d_iput() method is called). If there are other - references, then d_drop() is called instead - - d_add: add a dentry to its parents hash list and then calls - d_instantiate() - - d_instantiate: add a dentry to the alias hash list for the inode and - updates the "d_inode" member. The "i_count" member in the - inode structure should be set/incremented. If the inode - pointer is NULL, the dentry is called a "negative - dentry". This function is commonly called when an inode is - created for an existing negative dentry - - d_lookup: look up a dentry given its parent and path name component - It looks up the child of that given name from the dcache - hash table. If it is found, the reference count is incremented - and the dentry is returned. The caller must use dput() - to free the dentry when it finishes using it. - -Mount Options -============= - -Parsing options ---------------- - -On mount and remount the filesystem is passed a string containing a -comma separated list of mount options. The options can have either of -these forms: - - option - option=value - -The <linux/parser.h> header defines an API that helps parse these -options. There are plenty of examples on how to use it in existing -filesystems. - -Showing options ---------------- - -If a filesystem accepts mount options, it must define show_options() -to show all the currently active options. The rules are: - - - options MUST be shown which are not default or their values differ - from the default - - - options MAY be shown which are enabled by default or have their - default value - -Options used only internally between a mount helper and the kernel -(such as file descriptors), or which only have an effect during the -mounting (such as ones controlling the creation of a journal) are exempt -from the above rules. - -The underlying reason for the above rules is to make sure, that a -mount can be accurately replicated (e.g. umounting and mounting again) -based on the information found in /proc/mounts. - -Resources -========= - -(Note some of these resources are not up-to-date with the latest kernel - version.) - -Creating Linux virtual filesystems. 2002 - <http://lwn.net/Articles/13325/> - -The Linux Virtual File-system Layer by Neil Brown. 1999 - <http://www.cse.unsw.edu.au/~neilb/oss/linux-commentary/vfs.html> - -A tour of the Linux VFS by Michael K. Johnson. 1996 - <http://www.tldp.org/LDP/khg/HyperNews/get/fs/vfstour.html> - -A small trail through the Linux kernel by Andries Brouwer. 2001 - <http://www.win.tue.nl/~aeb/linux/vfs/trail.html> diff --git a/Documentation/filesystems/virtiofs.rst b/Documentation/filesystems/virtiofs.rst new file mode 100644 index 000000000000..4f338e3cb3f7 --- /dev/null +++ b/Documentation/filesystems/virtiofs.rst @@ -0,0 +1,60 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================== +virtiofs: virtio-fs host<->guest shared file system +=================================================== + +- Copyright (C) 2019 Red Hat, Inc. + +Introduction +============ +The virtiofs file system for Linux implements a driver for the paravirtualized +VIRTIO "virtio-fs" device for guest<->host file system sharing. It allows a +guest to mount a directory that has been exported on the host. + +Guests often require access to files residing on the host or remote systems. +Use cases include making files available to new guests during installation, +booting from a root file system located on the host, persistent storage for +stateless or ephemeral guests, and sharing a directory between guests. + +Although it is possible to use existing network file systems for some of these +tasks, they require configuration steps that are hard to automate and they +expose the storage network to the guest. The virtio-fs device was designed to +solve these problems by providing file system access without networking. + +Furthermore the virtio-fs device takes advantage of the co-location of the +guest and host to increase performance and provide semantics that are not +possible with network file systems. + +Usage +===== +Mount file system with tag ``myfs`` on ``/mnt``: + +.. code-block:: sh + + guest# mount -t virtiofs myfs /mnt + +Please see https://virtio-fs.gitlab.io/ for details on how to configure QEMU +and the virtiofsd daemon. + +Internals +========= +Since the virtio-fs device uses the FUSE protocol for file system requests, the +virtiofs file system for Linux is integrated closely with the FUSE file system +client. The guest acts as the FUSE client while the host acts as the FUSE +server. The /dev/fuse interface between the kernel and userspace is replaced +with the virtio-fs device interface. + +FUSE requests are placed into a virtqueue and processed by the host. The +response portion of the buffer is filled in by the host and the guest handles +the request completion. + +Mapping /dev/fuse to virtqueues requires solving differences in semantics +between /dev/fuse and virtqueues. Each time the /dev/fuse device is read, the +FUSE client may choose which request to transfer, making it possible to +prioritize certain requests over others. Virtqueues have queue semantics and +it is not possible to change the order of requests that have been enqueued. +This is especially important if the virtqueue becomes full since it is then +impossible to add high priority requests. In order to address this difference, +the virtio-fs device uses a "hiprio" virtqueue specifically for requests that +have priority over normal requests. diff --git a/Documentation/filesystems/xfs-delayed-logging-design.txt b/Documentation/filesystems/xfs-delayed-logging-design.txt index 2ce36439c09f..9a6dd289b17b 100644 --- a/Documentation/filesystems/xfs-delayed-logging-design.txt +++ b/Documentation/filesystems/xfs-delayed-logging-design.txt @@ -34,7 +34,7 @@ transaction: D A+B+C+D X+n+m+o <object written to disk> E E Y (> X+n+m+o) - F E+F Yٍ+p + F E+F Y+p In other words, each time an object is relogged, the new transaction contains the aggregation of all the previous changes currently held only in the log. diff --git a/Documentation/filesystems/xfs-self-describing-metadata.txt b/Documentation/filesystems/xfs-self-describing-metadata.txt index 68604e67a495..8db0121d0980 100644 --- a/Documentation/filesystems/xfs-self-describing-metadata.txt +++ b/Documentation/filesystems/xfs-self-describing-metadata.txt @@ -222,7 +222,7 @@ static void xfs_foo_read_verify( struct xfs_buf *bp) { - struct xfs_mount *mp = bp->b_target->bt_mount; + struct xfs_mount *mp = bp->b_mount; if ((xfs_sb_version_hascrc(&mp->m_sb) && !xfs_verify_cksum(bp->b_addr, BBTOB(bp->b_length), @@ -245,7 +245,7 @@ static bool xfs_foo_verify( struct xfs_buf *bp) { - struct xfs_mount *mp = bp->b_target->bt_mount; + struct xfs_mount *mp = bp->b_mount; struct xfs_ondisk_hdr *hdr = bp->b_addr; if (hdr->magic != cpu_to_be32(XFS_FOO_MAGIC)) @@ -272,7 +272,7 @@ static bool xfs_foo_verify( struct xfs_buf *bp) { - struct xfs_mount *mp = bp->b_target->bt_mount; + struct xfs_mount *mp = bp->b_mount; struct xfs_ondisk_hdr *hdr = bp->b_addr; if (hdr->magic == cpu_to_be32(XFS_FOO_CRC_MAGIC)) { @@ -297,7 +297,7 @@ static void xfs_foo_write_verify( struct xfs_buf *bp) { - struct xfs_mount *mp = bp->b_target->bt_mount; + struct xfs_mount *mp = bp->b_mount; struct xfs_buf_log_item *bip = bp->b_fspriv; if (!xfs_foo_verify(bp)) { diff --git a/Documentation/filesystems/xfs.txt b/Documentation/filesystems/xfs.txt deleted file mode 100644 index a5cbb5e0e3db..000000000000 --- a/Documentation/filesystems/xfs.txt +++ /dev/null @@ -1,470 +0,0 @@ - -The SGI XFS Filesystem -====================== - -XFS is a high performance journaling filesystem which originated -on the SGI IRIX platform. It is completely multi-threaded, can -support large files and large filesystems, extended attributes, -variable block sizes, is extent based, and makes extensive use of -Btrees (directories, extents, free space) to aid both performance -and scalability. - -Refer to the documentation at https://xfs.wiki.kernel.org/ -for further details. This implementation is on-disk compatible -with the IRIX version of XFS. - - -Mount Options -============= - -When mounting an XFS filesystem, the following options are accepted. -For boolean mount options, the names with the (*) suffix is the -default behaviour. - - allocsize=size - Sets the buffered I/O end-of-file preallocation size when - doing delayed allocation writeout (default size is 64KiB). - Valid values for this option are page size (typically 4KiB) - through to 1GiB, inclusive, in power-of-2 increments. - - The default behaviour is for dynamic end-of-file - preallocation size, which uses a set of heuristics to - optimise the preallocation size based on the current - allocation patterns within the file and the access patterns - to the file. Specifying a fixed allocsize value turns off - the dynamic behaviour. - - attr2 - noattr2 - The options enable/disable an "opportunistic" improvement to - be made in the way inline extended attributes are stored - on-disk. When the new form is used for the first time when - attr2 is selected (either when setting or removing extended - attributes) the on-disk superblock feature bit field will be - updated to reflect this format being in use. - - The default behaviour is determined by the on-disk feature - bit indicating that attr2 behaviour is active. If either - mount option it set, then that becomes the new default used - by the filesystem. - - CRC enabled filesystems always use the attr2 format, and so - will reject the noattr2 mount option if it is set. - - discard - nodiscard (*) - Enable/disable the issuing of commands to let the block - device reclaim space freed by the filesystem. This is - useful for SSD devices, thinly provisioned LUNs and virtual - machine images, but may have a performance impact. - - Note: It is currently recommended that you use the fstrim - application to discard unused blocks rather than the discard - mount option because the performance impact of this option - is quite severe. - - grpid/bsdgroups - nogrpid/sysvgroups (*) - These options define what group ID a newly created file - gets. When grpid is set, it takes the group ID of the - directory in which it is created; otherwise it takes the - fsgid of the current process, unless the directory has the - setgid bit set, in which case it takes the gid from the - parent directory, and also gets the setgid bit set if it is - a directory itself. - - filestreams - Make the data allocator use the filestreams allocation mode - across the entire filesystem rather than just on directories - configured to use it. - - ikeep - noikeep (*) - When ikeep is specified, XFS does not delete empty inode - clusters and keeps them around on disk. When noikeep is - specified, empty inode clusters are returned to the free - space pool. - - inode32 - inode64 (*) - When inode32 is specified, it indicates that XFS limits - inode creation to locations which will not result in inode - numbers with more than 32 bits of significance. - - When inode64 is specified, it indicates that XFS is allowed - to create inodes at any location in the filesystem, - including those which will result in inode numbers occupying - more than 32 bits of significance. - - inode32 is provided for backwards compatibility with older - systems and applications, since 64 bits inode numbers might - cause problems for some applications that cannot handle - large inode numbers. If applications are in use which do - not handle inode numbers bigger than 32 bits, the inode32 - option should be specified. - - - largeio - nolargeio (*) - If "nolargeio" is specified, the optimal I/O reported in - st_blksize by stat(2) will be as small as possible to allow - user applications to avoid inefficient read/modify/write - I/O. This is typically the page size of the machine, as - this is the granularity of the page cache. - - If "largeio" specified, a filesystem that was created with a - "swidth" specified will return the "swidth" value (in bytes) - in st_blksize. If the filesystem does not have a "swidth" - specified but does specify an "allocsize" then "allocsize" - (in bytes) will be returned instead. Otherwise the behaviour - is the same as if "nolargeio" was specified. - - logbufs=value - Set the number of in-memory log buffers. Valid numbers - range from 2-8 inclusive. - - The default value is 8 buffers. - - If the memory cost of 8 log buffers is too high on small - systems, then it may be reduced at some cost to performance - on metadata intensive workloads. The logbsize option below - controls the size of each buffer and so is also relevant to - this case. - - logbsize=value - Set the size of each in-memory log buffer. The size may be - specified in bytes, or in kilobytes with a "k" suffix. - Valid sizes for version 1 and version 2 logs are 16384 (16k) - and 32768 (32k). Valid sizes for version 2 logs also - include 65536 (64k), 131072 (128k) and 262144 (256k). The - logbsize must be an integer multiple of the log - stripe unit configured at mkfs time. - - The default value for for version 1 logs is 32768, while the - default value for version 2 logs is MAX(32768, log_sunit). - - logdev=device and rtdev=device - Use an external log (metadata journal) and/or real-time device. - An XFS filesystem has up to three parts: a data section, a log - section, and a real-time section. The real-time section is - optional, and the log section can be separate from the data - section or contained within it. - - noalign - Data allocations will not be aligned at stripe unit - boundaries. This is only relevant to filesystems created - with non-zero data alignment parameters (sunit, swidth) by - mkfs. - - norecovery - The filesystem will be mounted without running log recovery. - If the filesystem was not cleanly unmounted, it is likely to - be inconsistent when mounted in "norecovery" mode. - Some files or directories may not be accessible because of this. - Filesystems mounted "norecovery" must be mounted read-only or - the mount will fail. - - nouuid - Don't check for double mounted file systems using the file - system uuid. This is useful to mount LVM snapshot volumes, - and often used in combination with "norecovery" for mounting - read-only snapshots. - - noquota - Forcibly turns off all quota accounting and enforcement - within the filesystem. - - uquota/usrquota/uqnoenforce/quota - User disk quota accounting enabled, and limits (optionally) - enforced. Refer to xfs_quota(8) for further details. - - gquota/grpquota/gqnoenforce - Group disk quota accounting enabled and limits (optionally) - enforced. Refer to xfs_quota(8) for further details. - - pquota/prjquota/pqnoenforce - Project disk quota accounting enabled and limits (optionally) - enforced. Refer to xfs_quota(8) for further details. - - sunit=value and swidth=value - Used to specify the stripe unit and width for a RAID device - or a stripe volume. "value" must be specified in 512-byte - block units. These options are only relevant to filesystems - that were created with non-zero data alignment parameters. - - The sunit and swidth parameters specified must be compatible - with the existing filesystem alignment characteristics. In - general, that means the only valid changes to sunit are - increasing it by a power-of-2 multiple. Valid swidth values - are any integer multiple of a valid sunit value. - - Typically the only time these mount options are necessary if - after an underlying RAID device has had it's geometry - modified, such as adding a new disk to a RAID5 lun and - reshaping it. - - swalloc - Data allocations will be rounded up to stripe width boundaries - when the current end of file is being extended and the file - size is larger than the stripe width size. - - wsync - When specified, all filesystem namespace operations are - executed synchronously. This ensures that when the namespace - operation (create, unlink, etc) completes, the change to the - namespace is on stable storage. This is useful in HA setups - where failover must not result in clients seeing - inconsistent namespace presentation during or after a - failover event. - - -Deprecated Mount Options -======================== - - Name Removal Schedule - ---- ---------------- - - -Removed Mount Options -===================== - - Name Removed - ---- ------- - delaylog/nodelaylog v4.0 - ihashsize v4.0 - irixsgid v4.0 - osyncisdsync/osyncisosync v4.0 - barrier v4.19 - nobarrier v4.19 - - -sysctls -======= - -The following sysctls are available for the XFS filesystem: - - fs.xfs.stats_clear (Min: 0 Default: 0 Max: 1) - Setting this to "1" clears accumulated XFS statistics - in /proc/fs/xfs/stat. It then immediately resets to "0". - - fs.xfs.xfssyncd_centisecs (Min: 100 Default: 3000 Max: 720000) - The interval at which the filesystem flushes metadata - out to disk and runs internal cache cleanup routines. - - fs.xfs.filestream_centisecs (Min: 1 Default: 3000 Max: 360000) - The interval at which the filesystem ages filestreams cache - references and returns timed-out AGs back to the free stream - pool. - - fs.xfs.speculative_prealloc_lifetime - (Units: seconds Min: 1 Default: 300 Max: 86400) - The interval at which the background scanning for inodes - with unused speculative preallocation runs. The scan - removes unused preallocation from clean inodes and releases - the unused space back to the free pool. - - fs.xfs.error_level (Min: 0 Default: 3 Max: 11) - A volume knob for error reporting when internal errors occur. - This will generate detailed messages & backtraces for filesystem - shutdowns, for example. Current threshold values are: - - XFS_ERRLEVEL_OFF: 0 - XFS_ERRLEVEL_LOW: 1 - XFS_ERRLEVEL_HIGH: 5 - - fs.xfs.panic_mask (Min: 0 Default: 0 Max: 256) - Causes certain error conditions to call BUG(). Value is a bitmask; - OR together the tags which represent errors which should cause panics: - - XFS_NO_PTAG 0 - XFS_PTAG_IFLUSH 0x00000001 - XFS_PTAG_LOGRES 0x00000002 - XFS_PTAG_AILDELETE 0x00000004 - XFS_PTAG_ERROR_REPORT 0x00000008 - XFS_PTAG_SHUTDOWN_CORRUPT 0x00000010 - XFS_PTAG_SHUTDOWN_IOERROR 0x00000020 - XFS_PTAG_SHUTDOWN_LOGERROR 0x00000040 - XFS_PTAG_FSBLOCK_ZERO 0x00000080 - XFS_PTAG_VERIFIER_ERROR 0x00000100 - - This option is intended for debugging only. - - fs.xfs.irix_symlink_mode (Min: 0 Default: 0 Max: 1) - Controls whether symlinks are created with mode 0777 (default) - or whether their mode is affected by the umask (irix mode). - - fs.xfs.irix_sgid_inherit (Min: 0 Default: 0 Max: 1) - Controls files created in SGID directories. - If the group ID of the new file does not match the effective group - ID or one of the supplementary group IDs of the parent dir, the - ISGID bit is cleared if the irix_sgid_inherit compatibility sysctl - is set. - - fs.xfs.inherit_sync (Min: 0 Default: 1 Max: 1) - Setting this to "1" will cause the "sync" flag set - by the xfs_io(8) chattr command on a directory to be - inherited by files in that directory. - - fs.xfs.inherit_nodump (Min: 0 Default: 1 Max: 1) - Setting this to "1" will cause the "nodump" flag set - by the xfs_io(8) chattr command on a directory to be - inherited by files in that directory. - - fs.xfs.inherit_noatime (Min: 0 Default: 1 Max: 1) - Setting this to "1" will cause the "noatime" flag set - by the xfs_io(8) chattr command on a directory to be - inherited by files in that directory. - - fs.xfs.inherit_nosymlinks (Min: 0 Default: 1 Max: 1) - Setting this to "1" will cause the "nosymlinks" flag set - by the xfs_io(8) chattr command on a directory to be - inherited by files in that directory. - - fs.xfs.inherit_nodefrag (Min: 0 Default: 1 Max: 1) - Setting this to "1" will cause the "nodefrag" flag set - by the xfs_io(8) chattr command on a directory to be - inherited by files in that directory. - - fs.xfs.rotorstep (Min: 1 Default: 1 Max: 256) - In "inode32" allocation mode, this option determines how many - files the allocator attempts to allocate in the same allocation - group before moving to the next allocation group. The intent - is to control the rate at which the allocator moves between - allocation groups when allocating extents for new files. - -Deprecated Sysctls -================== - -None at present. - - -Removed Sysctls -=============== - - Name Removed - ---- ------- - fs.xfs.xfsbufd_centisec v4.0 - fs.xfs.age_buffer_centisecs v4.0 - - -Error handling -============== - -XFS can act differently according to the type of error found during its -operation. The implementation introduces the following concepts to the error -handler: - - -failure speed: - Defines how fast XFS should propagate an error upwards when a specific - error is found during the filesystem operation. It can propagate - immediately, after a defined number of retries, after a set time period, - or simply retry forever. - - -error classes: - Specifies the subsystem the error configuration will apply to, such as - metadata IO or memory allocation. Different subsystems will have - different error handlers for which behaviour can be configured. - - -error handlers: - Defines the behavior for a specific error. - -The filesystem behavior during an error can be set via sysfs files. Each -error handler works independently - the first condition met by an error handler -for a specific class will cause the error to be propagated rather than reset and -retried. - -The action taken by the filesystem when the error is propagated is context -dependent - it may cause a shut down in the case of an unrecoverable error, -it may be reported back to userspace, or it may even be ignored because -there's nothing useful we can with the error or anyone we can report it to (e.g. -during unmount). - -The configuration files are organized into the following hierarchy for each -mounted filesystem: - - /sys/fs/xfs/<dev>/error/<class>/<error>/ - -Where: - <dev> - The short device name of the mounted filesystem. This is the same device - name that shows up in XFS kernel error messages as "XFS(<dev>): ..." - - <class> - The subsystem the error configuration belongs to. As of 4.9, the defined - classes are: - - - "metadata": applies metadata buffer write IO - - <error> - The individual error handler configurations. - - -Each filesystem has "global" error configuration options defined in their top -level directory: - - /sys/fs/xfs/<dev>/error/ - - fail_at_unmount (Min: 0 Default: 1 Max: 1) - Defines the filesystem error behavior at unmount time. - - If set to a value of 1, XFS will override all other error configurations - during unmount and replace them with "immediate fail" characteristics. - i.e. no retries, no retry timeout. This will always allow unmount to - succeed when there are persistent errors present. - - If set to 0, the configured retry behaviour will continue until all - retries and/or timeouts have been exhausted. This will delay unmount - completion when there are persistent errors, and it may prevent the - filesystem from ever unmounting fully in the case of "retry forever" - handler configurations. - - Note: there is no guarantee that fail_at_unmount can be set while an - unmount is in progress. It is possible that the sysfs entries are - removed by the unmounting filesystem before a "retry forever" error - handler configuration causes unmount to hang, and hence the filesystem - must be configured appropriately before unmount begins to prevent - unmount hangs. - -Each filesystem has specific error class handlers that define the error -propagation behaviour for specific errors. There is also a "default" error -handler defined, which defines the behaviour for all errors that don't have -specific handlers defined. Where multiple retry constraints are configuredi for -a single error, the first retry configuration that expires will cause the error -to be propagated. The handler configurations are found in the directory: - - /sys/fs/xfs/<dev>/error/<class>/<error>/ - - max_retries (Min: -1 Default: Varies Max: INTMAX) - Defines the allowed number of retries of a specific error before - the filesystem will propagate the error. The retry count for a given - error context (e.g. a specific metadata buffer) is reset every time - there is a successful completion of the operation. - - Setting the value to "-1" will cause XFS to retry forever for this - specific error. - - Setting the value to "0" will cause XFS to fail immediately when the - specific error is reported. - - Setting the value to "N" (where 0 < N < Max) will make XFS retry the - operation "N" times before propagating the error. - - retry_timeout_seconds (Min: -1 Default: Varies Max: 1 day) - Define the amount of time (in seconds) that the filesystem is - allowed to retry its operations when the specific error is - found. - - Setting the value to "-1" will allow XFS to retry forever for this - specific error. - - Setting the value to "0" will cause XFS to fail immediately when the - specific error is reported. - - Setting the value to "N" (where 0 < N < Max) will allow XFS to retry the - operation for up to "N" seconds before propagating the error. - -Note: The default behaviour for a specific error handler is dependent on both -the class and error context. For example, the default values for -"metadata/ENODEV" are "0" rather than "-1" so that this error handler defaults -to "fail immediately" behaviour. This is done because ENODEV is a fatal, -unrecoverable error no matter how many times the metadata IO is retried. |