diff options
author | Dmitry Torokhov <dtor_core@ameritech.net> | 2005-09-09 20:14:47 -0500 |
---|---|---|
committer | Dmitry Torokhov <dtor_core@ameritech.net> | 2005-09-09 20:14:47 -0500 |
commit | d344c5e0856ad03278d8700b503762dbc8b86e12 (patch) | |
tree | a6d893a643470a3c2580a58f3228a55fa1fd1d82 /Documentation/filesystems | |
parent | 010988e888a0abbe7118635c1b33d049caae6b29 (diff) | |
parent | 87fc767b832ef5a681a0ff9d203c3289bc3be2bf (diff) | |
download | op-kernel-dev-d344c5e0856ad03278d8700b503762dbc8b86e12.zip op-kernel-dev-d344c5e0856ad03278d8700b503762dbc8b86e12.tar.gz |
Manual merge with Linus
Diffstat (limited to 'Documentation/filesystems')
-rw-r--r-- | Documentation/filesystems/files.txt | 123 | ||||
-rw-r--r-- | Documentation/filesystems/fuse.txt | 315 | ||||
-rw-r--r-- | Documentation/filesystems/ntfs.txt | 12 | ||||
-rw-r--r-- | Documentation/filesystems/proc.txt | 41 | ||||
-rw-r--r-- | Documentation/filesystems/relayfs.txt | 362 | ||||
-rw-r--r-- | Documentation/filesystems/sysfs.txt | 28 | ||||
-rw-r--r-- | Documentation/filesystems/v9fs.txt | 95 | ||||
-rw-r--r-- | Documentation/filesystems/vfs.txt | 435 |
8 files changed, 1276 insertions, 135 deletions
diff --git a/Documentation/filesystems/files.txt b/Documentation/filesystems/files.txt new file mode 100644 index 0000000..8c206f4 --- /dev/null +++ b/Documentation/filesystems/files.txt @@ -0,0 +1,123 @@ +File management in the Linux kernel +----------------------------------- + +This document describes how locking for files (struct file) +and file descriptor table (struct files) works. + +Up until 2.6.12, the file descriptor table has been protected +with a lock (files->file_lock) and reference count (files->count). +->file_lock protected accesses to all the file related fields +of the table. ->count was used for sharing the file descriptor +table between tasks cloned with CLONE_FILES flag. Typically +this would be the case for posix threads. As with the common +refcounting model in the kernel, the last task doing +a put_files_struct() frees the file descriptor (fd) table. +The files (struct file) themselves are protected using +reference count (->f_count). + +In the new lock-free model of file descriptor management, +the reference counting is similar, but the locking is +based on RCU. The file descriptor table contains multiple +elements - the fd sets (open_fds and close_on_exec, the +array of file pointers, the sizes of the sets and the array +etc.). In order for the updates to appear atomic to +a lock-free reader, all the elements of the file descriptor +table are in a separate structure - struct fdtable. +files_struct contains a pointer to struct fdtable through +which the actual fd table is accessed. Initially the +fdtable is embedded in files_struct itself. On a subsequent +expansion of fdtable, a new fdtable structure is allocated +and files->fdtab points to the new structure. The fdtable +structure is freed with RCU and lock-free readers either +see the old fdtable or the new fdtable making the update +appear atomic. Here are the locking rules for +the fdtable structure - + +1. All references to the fdtable must be done through + the files_fdtable() macro : + + struct fdtable *fdt; + + rcu_read_lock(); + + fdt = files_fdtable(files); + .... + if (n <= fdt->max_fds) + .... + ... + rcu_read_unlock(); + + files_fdtable() uses rcu_dereference() macro which takes care of + the memory barrier requirements for lock-free dereference. + The fdtable pointer must be read within the read-side + critical section. + +2. Reading of the fdtable as described above must be protected + by rcu_read_lock()/rcu_read_unlock(). + +3. For any update to the the fd table, files->file_lock must + be held. + +4. To look up the file structure given an fd, a reader + must use either fcheck() or fcheck_files() APIs. These + take care of barrier requirements due to lock-free lookup. + An example : + + struct file *file; + + rcu_read_lock(); + file = fcheck(fd); + if (file) { + ... + } + .... + rcu_read_unlock(); + +5. Handling of the file structures is special. Since the look-up + of the fd (fget()/fget_light()) are lock-free, it is possible + that look-up may race with the last put() operation on the + file structure. This is avoided using the rcuref APIs + on ->f_count : + + rcu_read_lock(); + file = fcheck_files(files, fd); + if (file) { + if (rcuref_inc_lf(&file->f_count)) + *fput_needed = 1; + else + /* Didn't get the reference, someone's freed */ + file = NULL; + } + rcu_read_unlock(); + .... + return file; + + rcuref_inc_lf() detects if refcounts is already zero or + goes to zero during increment. If it does, we fail + fget()/fget_light(). + +6. Since both fdtable and file structures can be looked up + lock-free, they must be installed using rcu_assign_pointer() + API. If they are looked up lock-free, rcu_dereference() + must be used. However it is advisable to use files_fdtable() + and fcheck()/fcheck_files() which take care of these issues. + +7. While updating, the fdtable pointer must be looked up while + holding files->file_lock. If ->file_lock is dropped, then + another thread expand the files thereby creating a new + fdtable and making the earlier fdtable pointer stale. + For example : + + spin_lock(&files->file_lock); + fd = locate_fd(files, file, start); + if (fd >= 0) { + /* locate_fd() may have expanded fdtable, load the ptr */ + fdt = files_fdtable(files); + FD_SET(fd, fdt->open_fds); + FD_CLR(fd, fdt->close_on_exec); + spin_unlock(&files->file_lock); + ..... + + Since locate_fd() can drop ->file_lock (and reacquire ->file_lock), + the fdtable pointer (fdt) must be loaded after locate_fd(). + diff --git a/Documentation/filesystems/fuse.txt b/Documentation/filesystems/fuse.txt new file mode 100644 index 0000000..6b5741e --- /dev/null +++ b/Documentation/filesystems/fuse.txt @@ -0,0 +1,315 @@ +Definitions +~~~~~~~~~~~ + +Userspace filesystem: + + A filesystem in which data and metadata are provided by an ordinary + userspace process. The filesystem can be accessed normally through + the kernel interface. + +Filesystem daemon: + + The process(es) providing the data and metadata of the filesystem. + +Non-privileged mount (or user mount): + + A userspace filesystem mounted by a non-privileged (non-root) user. + The filesystem daemon is running with the privileges of the mounting + user. NOTE: this is not the same as mounts allowed with the "user" + option in /etc/fstab, which is not discussed here. + +Mount owner: + + The user who does the mounting. + +User: + + The user who is performing filesystem operations. + +What is FUSE? +~~~~~~~~~~~~~ + +FUSE is a userspace filesystem framework. It consists of a kernel +module (fuse.ko), a userspace library (libfuse.*) and a mount utility +(fusermount). + +One of the most important features of FUSE is allowing secure, +non-privileged mounts. This opens up new possibilities for the use of +filesystems. A good example is sshfs: a secure network filesystem +using the sftp protocol. + +The userspace library and utilities are available from the FUSE +homepage: + + http://fuse.sourceforge.net/ + +Mount options +~~~~~~~~~~~~~ + +'fd=N' + + The file descriptor to use for communication between the userspace + filesystem and the kernel. The file descriptor must have been + obtained by opening the FUSE device ('/dev/fuse'). + +'rootmode=M' + + The file mode of the filesystem's root in octal representation. + +'user_id=N' + + The numeric user id of the mount owner. + +'group_id=N' + + The numeric group id of the mount owner. + +'default_permissions' + + By default FUSE doesn't check file access permissions, the + filesystem is free to implement it's access policy or leave it to + the underlying file access mechanism (e.g. in case of network + filesystems). This option enables permission checking, restricting + access based on file mode. This is option is usually useful + together with the 'allow_other' mount option. + +'allow_other' + + This option overrides the security measure restricting file access + to the user mounting the filesystem. This option is by default only + allowed to root, but this restriction can be removed with a + (userspace) configuration option. + +'max_read=N' + + With this option the maximum size of read operations can be set. + The default is infinite. Note that the size of read requests is + limited anyway to 32 pages (which is 128kbyte on i386). + +How do non-privileged mounts work? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Since the mount() system call is a privileged operation, a helper +program (fusermount) is needed, which is installed setuid root. + +The implication of providing non-privileged mounts is that the mount +owner must not be able to use this capability to compromise the +system. Obvious requirements arising from this are: + + A) mount owner should not be able to get elevated privileges with the + help of the mounted filesystem + + B) mount owner should not get illegitimate access to information from + other users' and the super user's processes + + C) mount owner should not be able to induce undesired behavior in + other users' or the super user's processes + +How are requirements fulfilled? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + A) The mount owner could gain elevated privileges by either: + + 1) creating a filesystem containing a device file, then opening + this device + + 2) creating a filesystem containing a suid or sgid application, + then executing this application + + The solution is not to allow opening device files and ignore + setuid and setgid bits when executing programs. To ensure this + fusermount always adds "nosuid" and "nodev" to the mount options + for non-privileged mounts. + + B) If another user is accessing files or directories in the + filesystem, the filesystem daemon serving requests can record the + exact sequence and timing of operations performed. This + information is otherwise inaccessible to the mount owner, so this + counts as an information leak. + + The solution to this problem will be presented in point 2) of C). + + C) There are several ways in which the mount owner can induce + undesired behavior in other users' processes, such as: + + 1) mounting a filesystem over a file or directory which the mount + owner could otherwise not be able to modify (or could only + make limited modifications). + + This is solved in fusermount, by checking the access + permissions on the mountpoint and only allowing the mount if + the mount owner can do unlimited modification (has write + access to the mountpoint, and mountpoint is not a "sticky" + directory) + + 2) Even if 1) is solved the mount owner can change the behavior + of other users' processes. + + i) It can slow down or indefinitely delay the execution of a + filesystem operation creating a DoS against the user or the + whole system. For example a suid application locking a + system file, and then accessing a file on the mount owner's + filesystem could be stopped, and thus causing the system + file to be locked forever. + + ii) It can present files or directories of unlimited length, or + directory structures of unlimited depth, possibly causing a + system process to eat up diskspace, memory or other + resources, again causing DoS. + + The solution to this as well as B) is not to allow processes + to access the filesystem, which could otherwise not be + monitored or manipulated by the mount owner. Since if the + mount owner can ptrace a process, it can do all of the above + without using a FUSE mount, the same criteria as used in + ptrace can be used to check if a process is allowed to access + the filesystem or not. + + Note that the ptrace check is not strictly necessary to + prevent B/2/i, it is enough to check if mount owner has enough + privilege to send signal to the process accessing the + filesystem, since SIGSTOP can be used to get a similar effect. + +I think these limitations are unacceptable? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If a sysadmin trusts the users enough, or can ensure through other +measures, that system processes will never enter non-privileged +mounts, it can relax the last limitation with a "user_allow_other" +config option. If this config option is set, the mounting user can +add the "allow_other" mount option which disables the check for other +users' processes. + +Kernel - userspace interface +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following diagram shows how a filesystem operation (in this +example unlink) is performed in FUSE. + +NOTE: everything in this description is greatly simplified + + | "rm /mnt/fuse/file" | FUSE filesystem daemon + | | + | | >sys_read() + | | >fuse_dev_read() + | | >request_wait() + | | [sleep on fc->waitq] + | | + | >sys_unlink() | + | >fuse_unlink() | + | [get request from | + | fc->unused_list] | + | >request_send() | + | [queue req on fc->pending] | + | [wake up fc->waitq] | [woken up] + | >request_wait_answer() | + | [sleep on req->waitq] | + | | <request_wait() + | | [remove req from fc->pending] + | | [copy req to read buffer] + | | [add req to fc->processing] + | | <fuse_dev_read() + | | <sys_read() + | | + | | [perform unlink] + | | + | | >sys_write() + | | >fuse_dev_write() + | | [look up req in fc->processing] + | | [remove from fc->processing] + | | [copy write buffer to req] + | [woken up] | [wake up req->waitq] + | | <fuse_dev_write() + | | <sys_write() + | <request_wait_answer() | + | <request_send() | + | [add request to | + | fc->unused_list] | + | <fuse_unlink() | + | <sys_unlink() | + +There are a couple of ways in which to deadlock a FUSE filesystem. +Since we are talking about unprivileged userspace programs, +something must be done about these. + +Scenario 1 - Simple deadlock +----------------------------- + + | "rm /mnt/fuse/file" | FUSE filesystem daemon + | | + | >sys_unlink("/mnt/fuse/file") | + | [acquire inode semaphore | + | for "file"] | + | >fuse_unlink() | + | [sleep on req->waitq] | + | | <sys_read() + | | >sys_unlink("/mnt/fuse/file") + | | [acquire inode semaphore + | | for "file"] + | | *DEADLOCK* + +The solution for this is to allow requests to be interrupted while +they are in userspace: + + | [interrupted by signal] | + | <fuse_unlink() | + | [release semaphore] | [semaphore acquired] + | <sys_unlink() | + | | >fuse_unlink() + | | [queue req on fc->pending] + | | [wake up fc->waitq] + | | [sleep on req->waitq] + +If the filesystem daemon was single threaded, this will stop here, +since there's no other thread to dequeue and execute the request. +In this case the solution is to kill the FUSE daemon as well. If +there are multiple serving threads, you just have to kill them as +long as any remain. + +Moral: a filesystem which deadlocks, can soon find itself dead. + +Scenario 2 - Tricky deadlock +---------------------------- + +This one needs a carefully crafted filesystem. It's a variation on +the above, only the call back to the filesystem is not explicit, +but is caused by a pagefault. + + | Kamikaze filesystem thread 1 | Kamikaze filesystem thread 2 + | | + | [fd = open("/mnt/fuse/file")] | [request served normally] + | [mmap fd to 'addr'] | + | [close fd] | [FLUSH triggers 'magic' flag] + | [read a byte from addr] | + | >do_page_fault() | + | [find or create page] | + | [lock page] | + | >fuse_readpage() | + | [queue READ request] | + | [sleep on req->waitq] | + | | [read request to buffer] + | | [create reply header before addr] + | | >sys_write(addr - headerlength) + | | >fuse_dev_write() + | | [look up req in fc->processing] + | | [remove from fc->processing] + | | [copy write buffer to req] + | | >do_page_fault() + | | [find or create page] + | | [lock page] + | | * DEADLOCK * + +Solution is again to let the the request be interrupted (not +elaborated further). + +An additional problem is that while the write buffer is being +copied to the request, the request must not be interrupted. This +is because the destination address of the copy may not be valid +after the request is interrupted. + +This is solved with doing the copy atomically, and allowing +interruption while the page(s) belonging to the write buffer are +faulted with get_user_pages(). The 'req->locked' flag indicates +when the copy is taking place, and interruption is delayed until +this flag is unset. + diff --git a/Documentation/filesystems/ntfs.txt b/Documentation/filesystems/ntfs.txt index eef4aca0..a5fbc8e 100644 --- a/Documentation/filesystems/ntfs.txt +++ b/Documentation/filesystems/ntfs.txt @@ -439,6 +439,18 @@ ChangeLog Note, a technical ChangeLog aimed at kernel hackers is in fs/ntfs/ChangeLog. +2.1.24: + - Support journals ($LogFile) which have been modified by chkdsk. This + means users can boot into Windows after we marked the volume dirty. + The Windows boot will run chkdsk and then reboot. The user can then + immediately boot into Linux rather than having to do a full Windows + boot first before rebooting into Linux and we will recognize such a + journal and empty it as it is clean by definition. + - Support journals ($LogFile) with only one restart page as well as + journals with two different restart pages. We sanity check both and + either use the only sane one or the more recent one of the two in the + case that both are valid. + - Lots of bug fixes and enhancements across the board. 2.1.23: - Stamp the user space journal, aka transaction log, aka $UsnJrnl, if it is present and active thus telling Windows and applications using diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt index 6c98f2b..d477356 100644 --- a/Documentation/filesystems/proc.txt +++ b/Documentation/filesystems/proc.txt @@ -133,6 +133,7 @@ Table 1-1: Process specific entries in /proc statm Process memory status information status Process status in human readable form wchan If CONFIG_KALLSYMS is set, a pre-decoded wchan + smaps Extension based on maps, presenting the rss size for each mapped file .............................................................................. For example, to get the status information of a process, all you have to do is @@ -1240,16 +1241,38 @@ swap-intensive. overcommit_memory ----------------- -This file contains one value. The following algorithm is used to decide if -there's enough memory: if the value of overcommit_memory is positive, then -there's always enough memory. This is a useful feature, since programs often -malloc() huge amounts of memory 'just in case', while they only use a small -part of it. Leaving this value at 0 will lead to the failure of such a huge -malloc(), when in fact the system has enough memory for the program to run. +Controls overcommit of system memory, possibly allowing processes +to allocate (but not use) more memory than is actually available. -On the other hand, enabling this feature can cause you to run out of memory -and thrash the system to death, so large and/or important servers will want to -set this value to 0. + +0 - Heuristic overcommit handling. Obvious overcommits of + address space are refused. Used for a typical system. It + ensures a seriously wild allocation fails while allowing + overcommit to reduce swap usage. root is allowed to + allocate slighly more memory in this mode. This is the + default. + +1 - Always overcommit. Appropriate for some scientific + applications. + +2 - Don't overcommit. The total address space commit + for the system is not permitted to exceed swap plus a + configurable percentage (default is 50) of physical RAM. + Depending on the percentage you use, in most situations + this means a process will not be killed while attempting + to use already-allocated memory but will receive errors + on memory allocation as appropriate. + +overcommit_ratio +---------------- + +Percentage of physical memory size to include in overcommit calculations +(see above.) + +Memory allocation limit = swapspace + physmem * (overcommit_ratio / 100) + + swapspace = total size of all swap areas + physmem = size of physical memory in system nr_hugepages and hugetlb_shm_group ---------------------------------- diff --git a/Documentation/filesystems/relayfs.txt b/Documentation/filesystems/relayfs.txt new file mode 100644 index 0000000..d24e1b0 --- /dev/null +++ b/Documentation/filesystems/relayfs.txt @@ -0,0 +1,362 @@ + +relayfs - a high-speed data relay filesystem +============================================ + +relayfs is a filesystem designed to provide an efficient mechanism for +tools and facilities to relay large and potentially sustained streams +of data from kernel space to user space. + +The main abstraction of relayfs is the 'channel'. A channel consists +of a set of per-cpu kernel buffers each represented by a file in the +relayfs filesystem. Kernel clients write into a channel using +efficient write functions which automatically log to the current cpu's +channel buffer. User space applications mmap() the per-cpu files and +retrieve the data as it becomes available. + +The format of the data logged into the channel buffers is completely +up to the relayfs client; relayfs does however provide hooks which +allow clients to impose some stucture on the buffer data. Nor does +relayfs implement any form of data filtering - this also is left to +the client. The purpose is to keep relayfs as simple as possible. + +This document provides an overview of the relayfs API. The details of +the function parameters are documented along with the functions in the +filesystem code - please see that for details. + +Semantics +========= + +Each relayfs channel has one buffer per CPU, each buffer has one or +more sub-buffers. Messages are written to the first sub-buffer until +it is too full to contain a new message, in which case it it is +written to the next (if available). Messages are never split across +sub-buffers. At this point, userspace can be notified so it empties +the first sub-buffer, while the kernel continues writing to the next. + +When notified that a sub-buffer is full, the kernel knows how many +bytes of it are padding i.e. unused. Userspace can use this knowledge +to copy only valid data. + +After copying it, userspace can notify the kernel that a sub-buffer +has been consumed. + +relayfs can operate in a mode where it will overwrite data not yet +collected by userspace, and not wait for it to consume it. + +relayfs itself does not provide for communication of such data between +userspace and kernel, allowing the kernel side to remain simple and not +impose a single interface on userspace. It does provide a separate +helper though, described below. + +klog, relay-app & librelay +========================== + +relayfs itself is ready to use, but to make things easier, two +additional systems are provided. klog is a simple wrapper to make +writing formatted text or raw data to a channel simpler, regardless of +whether a channel to write into exists or not, or whether relayfs is +compiled into the kernel or is configured as a module. relay-app is +the kernel counterpart of userspace librelay.c, combined these two +files provide glue to easily stream data to disk, without having to +bother with housekeeping. klog and relay-app can be used together, +with klog providing high-level logging functions to the kernel and +relay-app taking care of kernel-user control and disk-logging chores. + +It is possible to use relayfs without relay-app & librelay, but you'll +have to implement communication between userspace and kernel, allowing +both to convey the state of buffers (full, empty, amount of padding). + +klog, relay-app and librelay can be found in the relay-apps tarball on +http://relayfs.sourceforge.net + +The relayfs user space API +========================== + +relayfs implements basic file operations for user space access to +relayfs channel buffer data. Here are the file operations that are +available and some comments regarding their behavior: + +open() enables user to open an _existing_ buffer. + +mmap() results in channel buffer being mapped into the caller's + memory space. Note that you can't do a partial mmap - you must + map the entire file, which is NRBUF * SUBBUFSIZE. + +read() read the contents of a channel buffer. The bytes read are + 'consumed' by the reader i.e. they won't be available again + to subsequent reads. If the channel is being used in + no-overwrite mode (the default), it can be read at any time + even if there's an active kernel writer. If the channel is + being used in overwrite mode and there are active channel + writers, results may be unpredictable - users should make + sure that all logging to the channel has ended before using + read() with overwrite mode. + +poll() POLLIN/POLLRDNORM/POLLERR supported. User applications are + notified when sub-buffer boundaries are crossed. + +close() decrements the channel buffer's refcount. When the refcount + reaches 0 i.e. when no process or kernel client has the buffer + open, the channel buffer is freed. + + +In order for a user application to make use of relayfs files, the +relayfs filesystem must be mounted. For example, + + mount -t relayfs relayfs /mnt/relay + +NOTE: relayfs doesn't need to be mounted for kernel clients to create + or use channels - it only needs to be mounted when user space + applications need access to the buffer data. + + +The relayfs kernel API +====================== + +Here's a summary of the API relayfs provides to in-kernel clients: + + + channel management functions: + + relay_open(base_filename, parent, subbuf_size, n_subbufs, + callbacks) + relay_close(chan) + relay_flush(chan) + relay_reset(chan) + relayfs_create_dir(name, parent) + relayfs_remove_dir(dentry) + + channel management typically called on instigation of userspace: + + relay_subbufs_consumed(chan, cpu, subbufs_consumed) + + write functions: + + relay_write(chan, data, length) + __relay_write(chan, data, length) + relay_reserve(chan, length) + + callbacks: + + subbuf_start(buf, subbuf, prev_subbuf, prev_padding) + buf_mapped(buf, filp) + buf_unmapped(buf, filp) + + helper functions: + + relay_buf_full(buf) + subbuf_start_reserve(buf, length) + + +Creating a channel +------------------ + +relay_open() is used to create a channel, along with its per-cpu +channel buffers. Each channel buffer will have an associated file +created for it in the relayfs filesystem, which can be opened and +mmapped from user space if desired. The files are named +basename0...basenameN-1 where N is the number of online cpus, and by +default will be created in the root of the filesystem. If you want a +directory structure to contain your relayfs files, you can create it +with relayfs_create_dir() and pass the parent directory to +relay_open(). Clients are responsible for cleaning up any directory +structure they create when the channel is closed - use +relayfs_remove_dir() for that. + +The total size of each per-cpu buffer is calculated by multiplying the +number of sub-buffers by the sub-buffer size passed into relay_open(). +The idea behind sub-buffers is that they're basically an extension of +double-buffering to N buffers, and they also allow applications to +easily implement random-access-on-buffer-boundary schemes, which can +be important for some high-volume applications. The number and size +of sub-buffers is completely dependent on the application and even for +the same application, different conditions will warrant different +values for these parameters at different times. Typically, the right +values to use are best decided after some experimentation; in general, +though, it's safe to assume that having only 1 sub-buffer is a bad +idea - you're guaranteed to either overwrite data or lose events +depending on the channel mode being used. + +Channel 'modes' +--------------- + +relayfs channels can be used in either of two modes - 'overwrite' or +'no-overwrite'. The mode is entirely determined by the implementation +of the subbuf_start() callback, as described below. In 'overwrite' +mode, also known as 'flight recorder' mode, writes continuously cycle +around the buffer and will never fail, but will unconditionally +overwrite old data regardless of whether it's actually been consumed. +In no-overwrite mode, writes will fail i.e. data will be lost, if the +number of unconsumed sub-buffers equals the total number of +sub-buffers in the channel. It should be clear that if there is no +consumer or if the consumer can't consume sub-buffers fast enought, +data will be lost in either case; the only difference is whether data +is lost from the beginning or the end of a buffer. + +As explained above, a relayfs channel is made of up one or more +per-cpu channel buffers, each implemented as a circular buffer +subdivided into one or more sub-buffers. Messages are written into +the current sub-buffer of the channel's current per-cpu buffer via the +write functions described below. Whenever a message can't fit into +the current sub-buffer, because there's no room left for it, the +client is notified via the subbuf_start() callback that a switch to a +new sub-buffer is about to occur. The client uses this callback to 1) +initialize the next sub-buffer if appropriate 2) finalize the previous +sub-buffer if appropriate and 3) return a boolean value indicating +whether or not to actually go ahead with the sub-buffer switch. + +To implement 'no-overwrite' mode, the userspace client would provide +an implementation of the subbuf_start() callback something like the +following: + +static int subbuf_start(struct rchan_buf *buf, + void *subbuf, + void *prev_subbuf, + unsigned int prev_padding) +{ + if (prev_subbuf) + *((unsigned *)prev_subbuf) = prev_padding; + + if (relay_buf_full(buf)) + return 0; + + subbuf_start_reserve(buf, sizeof(unsigned int)); + + return 1; +} + +If the current buffer is full i.e. all sub-buffers remain unconsumed, +the callback returns 0 to indicate that the buffer switch should not +occur yet i.e. until the consumer has had a chance to read the current +set of ready sub-buffers. For the relay_buf_full() function to make +sense, the consumer is reponsible for notifying relayfs when +sub-buffers have been consumed via relay_subbufs_consumed(). Any +subsequent attempts to write into the buffer will again invoke the +subbuf_start() callback with the same parameters; only when the +consumer has consumed one or more of the ready sub-buffers will +relay_buf_full() return 0, in which case the buffer switch can +continue. + +The implementation of the subbuf_start() callback for 'overwrite' mode +would be very similar: + +static int subbuf_start(struct rchan_buf *buf, + void *subbuf, + void *prev_subbuf, + unsigned int prev_padding) +{ + if (prev_subbuf) + *((unsigned *)prev_subbuf) = prev_padding; + + subbuf_start_reserve(buf, sizeof(unsigned int)); + + return 1; +} + +In this case, the relay_buf_full() check is meaningless and the +callback always returns 1, causing the buffer switch to occur +unconditionally. It's also meaningless for the client to use the +relay_subbufs_consumed() function in this mode, as it's never +consulted. + +The default subbuf_start() implementation, used if the client doesn't +define any callbacks, or doesn't define the subbuf_start() callback, +implements the simplest possible 'no-overwrite' mode i.e. it does +nothing but return 0. + +Header information can be reserved at the beginning of each sub-buffer +by calling the subbuf_start_reserve() helper function from within the +subbuf_start() callback. This reserved area can be used to store +whatever information the client wants. In the example above, room is +reserved in each sub-buffer to store the padding count for that +sub-buffer. This is filled in for the previous sub-buffer in the +subbuf_start() implementation; the padding value for the previous +sub-buffer is passed into the subbuf_start() callback along with a +pointer to the previous sub-buffer, since the padding value isn't +known until a sub-buffer is filled. The subbuf_start() callback is +also called for the first sub-buffer when the channel is opened, to +give the client a chance to reserve space in it. In this case the +previous sub-buffer pointer passed into the callback will be NULL, so +the client should check the value of the prev_subbuf pointer before +writing into the previous sub-buffer. + +Writing to a channel +-------------------- + +kernel clients write data into the current cpu's channel buffer using +relay_write() or __relay_write(). relay_write() is the main logging +function - it uses local_irqsave() to protect the buffer and should be +used if you might be logging from interrupt context. If you know +you'll never be logging from interrupt context, you can use +__relay_write(), which only disables preemption. These functions +don't return a value, so you can't determine whether or not they +failed - the assumption is that you wouldn't want to check a return +value in the fast logging path anyway, and that they'll always succeed +unless the buffer is full and no-overwrite mode is being used, in +which case you can detect a failed write in the subbuf_start() +callback by calling the relay_buf_full() helper function. + +relay_reserve() is used to reserve a slot in a channel buffer which +can be written to later. This would typically be used in applications +that need to write directly into a channel buffer without having to +stage data in a temporary buffer beforehand. Because the actual write +may not happen immediately after the slot is reserved, applications +using relay_reserve() can keep a count of the number of bytes actually +written, either in space reserved in the sub-buffers themselves or as +a separate array. See the 'reserve' example in the relay-apps tarball +at http://relayfs.sourceforge.net for an example of how this can be +done. Because the write is under control of the client and is +separated from the reserve, relay_reserve() doesn't protect the buffer +at all - it's up to the client to provide the appropriate +synchronization when using relay_reserve(). + +Closing a channel +----------------- + +The client calls relay_close() when it's finished using the channel. +The channel and its associated buffers are destroyed when there are no +longer any references to any of the channel buffers. relay_flush() +forces a sub-buffer switch on all the channel buffers, and can be used +to finalize and process the last sub-buffers before the channel is +closed. + +Misc +---- + +Some applications may want to keep a channel around and re-use it +rather than open and close a new channel for each use. relay_reset() +can be used for this purpose - it resets a channel to its initial +state without reallocating channel buffer memory or destroying +existing mappings. It should however only be called when it's safe to +do so i.e. when the channel isn't currently being written to. + +Finally, there are a couple of utility callbacks that can be used for +different purposes. buf_mapped() is called whenever a channel buffer +is mmapped from user space and buf_unmapped() is called when it's +unmapped. The client can use this notification to trigger actions +within the kernel application, such as enabling/disabling logging to +the channel. + + +Resources +========= + +For news, example code, mailing list, etc. see the relayfs homepage: + + http://relayfs.sourceforge.net + + +Credits +======= + +The ideas and specs for relayfs came about as a result of discussions +on tracing involving the following: + +Michel Dagenais <michel.dagenais@polymtl.ca> +Richard Moore <richardj_moore@uk.ibm.com> +Bob Wisniewski <bob@watson.ibm.com> +Karim Yaghmour <karim@opersys.com> +Tom Zanussi <zanussi@us.ibm.com> + +Also thanks to Hubertus Franke for a lot of useful suggestions and bug +reports. diff --git a/Documentation/filesystems/sysfs.txt b/Documentation/filesystems/sysfs.txt index dc27659..c8bce82 100644 --- a/Documentation/filesystems/sysfs.txt +++ b/Documentation/filesystems/sysfs.txt @@ -90,7 +90,7 @@ void device_remove_file(struct device *, struct device_attribute *); It also defines this helper for defining device attributes: -#define DEVICE_ATTR(_name,_mode,_show,_store) \ +#define DEVICE_ATTR(_name, _mode, _show, _store) \ struct device_attribute dev_attr_##_name = { \ .attr = {.name = __stringify(_name) , .mode = _mode }, \ .show = _show, \ @@ -99,14 +99,14 @@ struct device_attribute dev_attr_##_name = { \ For example, declaring -static DEVICE_ATTR(foo,0644,show_foo,store_foo); +static DEVICE_ATTR(foo, S_IWUSR | S_IRUGO, show_foo, store_foo); is equivalent to doing: static struct device_attribute dev_attr_foo = { .attr = { .name = "foo", - .mode = 0644, + .mode = S_IWUSR | S_IRUGO, }, .show = show_foo, .store = store_foo, @@ -121,8 +121,8 @@ set of sysfs operations for forwarding read and write calls to the show and store methods of the attribute owners. struct sysfs_ops { - ssize_t (*show)(struct kobject *, struct attribute *,char *); - ssize_t (*store)(struct kobject *,struct attribute *,const char *); + ssize_t (*show)(struct kobject *, struct attribute *, char *); + ssize_t (*store)(struct kobject *, struct attribute *, const char *); }; [ Subsystems should have already defined a struct kobj_type as a @@ -137,7 +137,7 @@ calls the associated methods. To illustrate: -#define to_dev_attr(_attr) container_of(_attr,struct device_attribute,attr) +#define to_dev_attr(_attr) container_of(_attr, struct device_attribute, attr) #define to_dev(d) container_of(d, struct device, kobj) static ssize_t @@ -148,7 +148,7 @@ dev_attr_show(struct kobject * kobj, struct attribute * attr, char * buf) ssize_t ret = 0; if (dev_attr->show) - ret = dev_attr->show(dev,buf); + ret = dev_attr->show(dev, buf); return ret; } @@ -216,16 +216,16 @@ A very simple (and naive) implementation of a device attribute is: static ssize_t show_name(struct device *dev, struct device_attribute *attr, char *buf) { - return sprintf(buf,"%s\n",dev->name); + return snprintf(buf, PAGE_SIZE, "%s\n", dev->name); } static ssize_t store_name(struct device * dev, const char * buf) { - sscanf(buf,"%20s",dev->name); - return strlen(buf); + sscanf(buf, "%20s", dev->name); + return strnlen(buf, PAGE_SIZE); } -static DEVICE_ATTR(name,S_IRUGO,show_name,store_name); +static DEVICE_ATTR(name, S_IRUGO, show_name, store_name); (Note that the real implementation doesn't allow userspace to set the @@ -290,7 +290,7 @@ struct device_attribute { Declaring: -DEVICE_ATTR(_name,_str,_mode,_show,_store); +DEVICE_ATTR(_name, _str, _mode, _show, _store); Creation/Removal: @@ -310,7 +310,7 @@ struct bus_attribute { Declaring: -BUS_ATTR(_name,_mode,_show,_store) +BUS_ATTR(_name, _mode, _show, _store) Creation/Removal: @@ -331,7 +331,7 @@ struct driver_attribute { Declaring: -DRIVER_ATTR(_name,_mode,_show,_store) +DRIVER_ATTR(_name, _mode, _show, _store) Creation/Removal: diff --git a/Documentation/filesystems/v9fs.txt b/Documentation/filesystems/v9fs.txt new file mode 100644 index 0000000..4e92feb --- /dev/null +++ b/Documentation/filesystems/v9fs.txt @@ -0,0 +1,95 @@ + V9FS: 9P2000 for Linux + ====================== + +ABOUT +===== + +v9fs is a Unix implementation of the Plan 9 9p remote filesystem protocol. + +This software was originally developed by Ron Minnich <rminnich@lanl.gov> +and Maya Gokhale <maya@lanl.gov>. Additional development by Greg Watson +<gwatson@lanl.gov> and most recently Eric Van Hensbergen +<ericvh@gmail.com> and Latchesar Ionkov <lucho@ionkov.net>. + +USAGE +===== + +For remote file server: + + mount -t 9P 10.10.1.2 /mnt/9 + +For Plan 9 From User Space applications (http://swtch.com/plan9) + + mount -t 9P `namespace`/acme /mnt/9 -o proto=unix,name=$USER + +OPTIONS +======= + + proto=name select an alternative transport. Valid options are + currently: + unix - specifying a named pipe mount point + tcp - specifying a normal TCP/IP connection + fd - used passed file descriptors for connection + (see rfdno and wfdno) + + name=name user name to attempt mount as on the remote server. The + server may override or ignore this value. Certain user + names may require authentication. + + aname=name aname specifies the file tree to access when the server is + offering several exported file systems. + + debug=n specifies debug level. The debug level is a bitmask. + 0x01 = display verbose error messages + 0x02 = developer debug (DEBUG_CURRENT) + 0x04 = display 9P trace + 0x08 = display VFS trace + 0x10 = display Marshalling debug + 0x20 = display RPC debug + 0x40 = display transport debug + 0x80 = display allocation debug + + rfdno=n the file descriptor for reading with proto=fd + + wfdno=n the file descriptor for writing with proto=fd + + maxdata=n the number of bytes to use for 9P packet payload (msize) + + port=n port to connect to on the remote server + + timeout=n request timeouts (in ms) (default 60000ms) + + noextend force legacy mode (no 9P2000.u semantics) + + uid attempt to mount as a particular uid + + gid attempt to mount with a particular gid + + afid security channel - used by Plan 9 authentication protocols + + nodevmap do not map special files - represent them as normal files. + This can be used to share devices/named pipes/sockets between + hosts. This functionality will be expanded in later versions. + +RESOURCES +========= + +The Linux version of the 9P server, along with some client-side utilities +can be found at http://v9fs.sf.net (along with a CVS repository of the +development branch of this module). There are user and developer mailing +lists here, as well as a bug-tracker. + +For more information on the Plan 9 Operating System check out +http://plan9.bell-labs.com/plan9 + +For information on Plan 9 from User Space (Plan 9 applications and libraries +ported to Linux/BSD/OSX/etc) check out http://swtch.com/plan9 + + +STATUS +====== + +The 2.6 kernel support is working on PPC and x86. + +PLEASE USE THE SOURCEFORGE BUG-TRACKER TO REPORT PROBLEMS. + diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt index 3f318dd..f042c12 100644 --- a/Documentation/filesystems/vfs.txt +++ b/Documentation/filesystems/vfs.txt @@ -1,35 +1,27 @@ -/* -*- auto-fill -*- */ - Overview of the Virtual File System + Overview of the Linux Virtual File System - Richard Gooch <rgooch@atnf.csiro.au> + Original author: Richard Gooch <rgooch@atnf.csiro.au> - 5-JUL-1999 + Last updated on August 25, 2005 + Copyright (C) 1999 Richard Gooch + Copyright (C) 2005 Pekka Enberg -Conventions used in this document <section> -================================= + This file is released under the GPLv2. -Each section in this document will have the string "<section>" at the -right-hand side of the section title. Each subsection will have -"<subsection>" at the right-hand side. These strings are meant to make -it easier to search through the document. -NOTE that the master copy of this document is available online at: -http://www.atnf.csiro.au/~rgooch/linux/docs/vfs.txt - - -What is it? <section> +What is it? =========== The Virtual File System (otherwise known as the Virtual Filesystem Switch) is the software layer in the kernel that provides the filesystem interface to userspace programs. It also provides an abstraction within the kernel which allows different filesystem -implementations to co-exist. +implementations to coexist. -A Quick Look At How It Works <section> +A Quick Look At How It Works ============================ In this section I'll briefly describe how things work, before @@ -38,7 +30,8 @@ when user programs open and manipulate files, and then look from the other view which is how a filesystem is supported and subsequently mounted. -Opening a File <subsection> + +Opening a File -------------- The VFS implements the open(2), stat(2), chmod(2) and similar system @@ -77,7 +70,7 @@ back to userspace. Opening a file requires another operation: allocation of a file structure (this is the kernel-side implementation of file -descriptors). The freshly allocated file structure is initialised with +descriptors). The freshly allocated file structure is initialized with a pointer to the dentry and a set of file operation member functions. These are taken from the inode data. The open() file method is then called so the specific filesystem implementation can do it's work. You @@ -102,7 +95,8 @@ filesystem or driver code at the same time, on different processors. You should ensure that access to shared resources is protected by appropriate locks. -Registering and Mounting a Filesystem <subsection> + +Registering and Mounting a Filesystem ------------------------------------- If you want to support a new kind of filesystem in the kernel, all you @@ -123,17 +117,21 @@ updated to point to the root inode for the new filesystem. It's now time to look at things in more detail. -struct file_system_type <section> +struct file_system_type ======================= -This describes the filesystem. As of kernel 2.1.99, the following +This describes the filesystem. As of kernel 2.6.13, the following members are defined: struct file_system_type { const char *name; int fs_flags; - struct super_block *(*read_super) (struct super_block *, void *, int); - struct file_system_type * next; + struct super_block *(*get_sb) (struct file_system_type *, int, + const char *, void *); + void (*kill_sb) (struct super_block *); + struct module *owner; + struct file_system_type * next; + struct list_head fs_supers; }; name: the name of the filesystem type, such as "ext2", "iso9660", @@ -141,51 +139,97 @@ struct file_system_type { fs_flags: various flags (i.e. FS_REQUIRES_DEV, FS_NO_DCACHE, etc.) - read_super: the method to call when a new instance of this + get_sb: the method to call when a new instance of this filesystem should be mounted - next: for internal VFS use: you should initialise this to NULL + kill_sb: the method to call when an instance of this filesystem + should be unmounted + + owner: for internal VFS use: you should initialize this to THIS_MODULE in + most cases. -The read_super() method has the following arguments: + next: for internal VFS use: you should initialize this to NULL + +The get_sb() method has the following arguments: struct super_block *sb: the superblock structure. This is partially - initialised by the VFS and the rest must be initialised by the - read_super() method + initialized by the VFS and the rest must be initialized by the + get_sb() method + + int flags: mount flags + + const char *dev_name: the device name we are mounting. void *data: arbitrary mount options, usually comes as an ASCII string int silent: whether or not to be silent on error -The read_super() method must determine if the block device specified +The get_sb() method must determine if the block device specified in the superblock contains a filesystem of the type the method supports. On success the method returns the superblock pointer, on failure it returns NULL. The most interesting member of the superblock structure that the -read_super() method fills in is the "s_op" field. This is a pointer to +get_sb() method fills in is the "s_op" field. This is a pointer to a "struct super_operations" which describes the next level of the filesystem implementation. +Usually, a filesystem uses generic one of the generic get_sb() +implementations and provides a fill_super() method instead. The +generic methods are: + + get_sb_bdev: mount a filesystem residing on a block device -struct super_operations <section> + get_sb_nodev: mount a filesystem that is not backed by a device + + get_sb_single: mount a filesystem which shares the instance between + all mounts + +A fill_super() method implementation has the following arguments: + + struct super_block *sb: the superblock structure. The method fill_super() + must initialize this properly. + + void *data: arbitrary mount options, usually comes as an ASCII + string + + int silent: whether or not to be silent on error + + +struct super_operations ======================= This describes how the VFS can manipulate the superblock of your -filesystem. As of kernel 2.1.99, the following members are defined: +filesystem. As of kernel 2.6.13, the following members are defined: struct super_operations { - void (*read_inode) (struct inode *); - int (*write_inode) (struct inode *, int); - void (*put_inode) (struct inode *); - void (*drop_inode) (struct inode *); - void (*delete_inode) (struct inode *); - int (*notify_change) (struct dentry *, struct iattr *); - void (*put_super) (struct super_block *); - void (*write_super) (struct super_block *); - int (*statfs) (struct super_block *, struct statfs *, int); - int (*remount_fs) (struct super_block *, int *, char *); - void (*clear_inode) (struct inode *); + struct inode *(*alloc_inode)(struct super_block *sb); + void (*destroy_inode)(struct inode *); + + void (*read_inode) (struct inode *); + + void (*dirty_inode) (struct inode *); + int (*write_inode) (struct inode *, int); + void (*put_inode) (struct inode *); + void (*drop_inode) (struct inode *); + void (*delete_inode) (struct inode *); + void (*put_super) (struct super_block *); + void (*write_super) (struct super_block *); + int (*sync_fs)(struct super_block *sb, int wait); + void (*write_super_lockfs) (struct super_block *); + void (*unlockfs) (struct super_block *); + int (*statfs) (struct super_block *, struct kstatfs *); + int (*remount_fs) (struct super_block *, int *, char *); + void (*clear_inode) (struct inode *); + void (*umount_begin) (struct super_block *); + + void (*sync_inodes) (struct super_block *sb, + struct writeback_control *wbc); + int (*show_options)(struct seq_file *, struct vfsmount *); + + ssize_t (*quota_read)(struct super_block *, int, char *, size_t, loff_t); + ssize_t (*quota_write)(struct super_block *, int, const char *, size_t, loff_t); }; All methods are called without any locks being held, unless otherwise @@ -193,43 +237,62 @@ noted. This means that most methods can block safely. All methods are only called from a process context (i.e. not from an interrupt handler or bottom half). + alloc_inode: this method is called by inode_alloc() to allocate memory + for struct inode and initialize it. + + destroy_inode: this method is called by destroy_inode() to release + resources allocated for struct inode. + read_inode: this method is called to read a specific inode from the - mounted filesystem. The "i_ino" member in the "struct inode" - will be initialised by the VFS to indicate which inode to - read. Other members are filled in by this method + mounted filesystem. The i_ino member in the struct inode is + initialized by the VFS to indicate which inode to read. Other + members are filled in by this method. + + You can set this to NULL and use iget5_locked() instead of iget() + to read inodes. This is necessary for filesystems for which the + inode number is not sufficient to identify an inode. + + dirty_inode: this method is called by the VFS to mark an inode dirty. write_inode: this method is called when the VFS needs to write an inode to disc. The second parameter indicates whether the write should be synchronous or not, not all filesystems check this flag. put_inode: called when the VFS inode is removed from the inode - cache. This method is optional + cache. drop_inode: called when the last access to the inode is dropped, with the inode_lock spinlock held. - This method should be either NULL (normal unix filesystem + This method should be either NULL (normal UNIX filesystem semantics) or "generic_delete_inode" (for filesystems that do not want to cache inodes - causing "delete_inode" to always be called regardless of the value of i_nlink) - The "generic_delete_inode()" behaviour is equivalent to the + The "generic_delete_inode()" behavior is equivalent to the old practice of using "force_delete" in the put_inode() case, but does not have the races that the "force_delete()" approach had. delete_inode: called when the VFS wants to delete an inode - notify_change: called when VFS inode attributes are changed. If this - is NULL the VFS falls back to the write_inode() method. This - is called with the kernel lock held - put_super: called when the VFS wishes to free the superblock (i.e. unmount). This is called with the superblock lock held write_super: called when the VFS superblock needs to be written to disc. This method is optional + sync_fs: called when VFS is writing out all dirty data associated with + a superblock. The second parameter indicates whether the method + should wait until the write out has been completed. Optional. + + write_super_lockfs: called when VFS is locking a filesystem and forcing + it into a consistent state. This function is currently used by the + Logical Volume Manager (LVM). + + unlockfs: called when VFS is unlocking a filesystem and making it writable + again. + statfs: called when the VFS needs to get filesystem statistics. This is called with the kernel lock held @@ -238,21 +301,31 @@ or bottom half). clear_inode: called then the VFS clears the inode. Optional + umount_begin: called when the VFS is unmounting a filesystem. + + sync_inodes: called when the VFS is writing out dirty data associated with + a superblock. + + show_options: called by the VFS to show mount options for /proc/<pid>/mounts. + + quota_read: called by the VFS to read from filesystem quota file. + + quota_write: called by the VFS to write to filesystem quota file. + The read_inode() method is responsible for filling in the "i_op" field. This is a pointer to a "struct inode_operations" which describes the methods that can be performed on individual inodes. -struct inode_operations <section> +struct inode_operations ======================= This describes how the VFS can manipulate an inode in your -filesystem. As of kernel 2.1.99, the following members are defined: +filesystem. As of kernel 2.6.13, the following members are defined: struct inode_operations { - struct file_operations * default_file_ops; - int (*create) (struct inode *,struct dentry *,int); - int (*lookup) (struct inode *,struct dentry *); + int (*create) (struct inode *,struct dentry *,int, struct nameidata *); + struct dentry * (*lookup) (struct inode *,struct dentry *, struct nameidata *); int (*link) (struct dentry *,struct inode *,struct dentry *); int (*unlink) (struct inode *,struct dentry *); int (*symlink) (struct inode *,struct dentry *,const char *); @@ -261,25 +334,22 @@ struct inode_operations { int (*mknod) (struct inode *,struct dentry *,int,dev_t); int (*rename) (struct inode *, struct dentry *, struct inode *, struct dentry *); - int (*readlink) (struct dentry *, char *,int); - struct dentry * (*follow_link) (struct dentry *, struct dentry *); - int (*readpage) (struct file *, struct page *); - int (*writepage) (struct page *page, struct writeback_control *wbc); - int (*bmap) (struct inode *,int); + int (*readlink) (struct dentry *, char __user *,int); + void * (*follow_link) (struct dentry *, struct nameidata *); + void (*put_link) (struct dentry *, struct nameidata *, void *); void (*truncate) (struct inode *); - int (*permission) (struct inode *, int); - int (*smap) (struct inode *,int); - int (*updatepage) (struct file *, struct page *, const char *, - unsigned long, unsigned int, int); - int (*revalidate) (struct dentry *); + int (*permission) (struct inode *, int, struct nameidata *); + int (*setattr) (struct dentry *, struct iattr *); + int (*getattr) (struct vfsmount *mnt, struct dentry *, struct kstat *); + int (*setxattr) (struct dentry *, const char *,const void *,size_t,int); + ssize_t (*getxattr) (struct dentry *, const char *, void *, size_t); + ssize_t (*listxattr) (struct dentry *, char *, size_t); + int (*removexattr) (struct dentry *, const char *); }; Again, all methods are called without any locks being held, unless otherwise noted. - default_file_ops: this is a pointer to a "struct file_operations" - which describes how to open and then manipulate open files - create: called by the open(2) and creat(2) system calls. Only required if you want to support regular files. The dentry you get should not have an inode (i.e. it should be a negative @@ -328,31 +398,143 @@ otherwise noted. you want to support reading symbolic links follow_link: called by the VFS to follow a symbolic link to the - inode it points to. Only required if you want to support - symbolic links + inode it points to. Only required if you want to support + symbolic links. This function returns a void pointer cookie + that is passed to put_link(). + + put_link: called by the VFS to release resources allocated by + follow_link(). The cookie returned by follow_link() is passed to + to this function as the last parameter. It is used by filesystems + such as NFS where page cache is not stable (i.e. page that was + installed when the symbolic link walk started might not be in the + page cache at the end of the walk). + + truncate: called by the VFS to change the size of a file. The i_size + field of the inode is set to the desired size by the VFS before + this function is called. This function is called by the truncate(2) + system call and related functionality. + + permission: called by the VFS to check for access rights on a POSIX-like + filesystem. + + setattr: called by the VFS to set attributes for a file. This function is + called by chmod(2) and related system calls. + + getattr: called by the VFS to get attributes of a file. This function is + called by stat(2) and related system calls. + + setxattr: called by the VFS to set an extended attribute for a file. + Extended attribute is a name:value pair associated with an inode. This + function is called by setxattr(2) system call. + + getxattr: called by the VFS to retrieve the value of an extended attribute + name. This function is called by getxattr(2) function call. + + listxattr: called by the VFS to list all extended attributes for a given + file. This function is called by listxattr(2) system call. + + removexattr: called by the VFS to remove an extended attribute from a file. + This function is called by removexattr(2) system call. + + +struct address_space_operations +=============================== + +This describes how the VFS can manipulate mapping of a file to page cache in +your filesystem. As of kernel 2.6.13, the following members are defined: + +struct address_space_operations { + int (*writepage)(struct page *page, struct writeback_control *wbc); + int (*readpage)(struct file *, struct page *); + int (*sync_page)(struct page *); + int (*writepages)(struct address_space *, struct writeback_control *); + int (*set_page_dirty)(struct page *page); + int (*readpages)(struct file *filp, struct address_space *mapping, + struct list_head *pages, unsigned nr_pages); + int (*prepare_write)(struct file *, struct page *, unsigned, unsigned); + int (*commit_write)(struct file *, struct page *, unsigned, unsigned); + sector_t (*bmap)(struct address_space *, sector_t); + int (*invalidatepage) (struct page *, unsigned long); + int (*releasepage) (struct page *, int); + ssize_t (*direct_IO)(int, struct kiocb *, const struct iovec *iov, + loff_t offset, unsigned long nr_segs); + struct page* (*get_xip_page)(struct address_space *, sector_t, + int); +}; + + writepage: called by the VM write a dirty page to backing store. + + readpage: called by the VM to read a page from backing store. + + sync_page: called by the VM to notify the backing store to perform all + queued I/O operations for a page. I/O operations for other pages + associated with this address_space object may also be performed. + + writepages: called by the VM to write out pages associated with the + address_space object. + + set_page_dirty: called by the VM to set a page dirty. + + readpages: called by the VM to read pages associated with the address_space + object. + prepare_write: called by the generic write path in VM to set up a write + request for a page. -struct file_operations <section> + commit_write: called by the generic write path in VM to write page to + its backing store. + + bmap: called by the VFS to map a logical block offset within object to + physical block number. This method is use by for the legacy FIBMAP + ioctl. Other uses are discouraged. + + invalidatepage: called by the VM on truncate to disassociate a page from its + address_space mapping. + + releasepage: called by the VFS to release filesystem specific metadata from + a page. + + direct_IO: called by the VM for direct I/O writes and reads. + + get_xip_page: called by the VM to translate a block number to a page. + The page is valid until the corresponding filesystem is unmounted. + Filesystems that want to use execute-in-place (XIP) need to implement + it. An example implementation can be found in fs/ext2/xip.c. + + +struct file_operations ====================== This describes how the VFS can manipulate an open file. As of kernel -2.1.99, the following members are defined: +2.6.13, the following members are defined: struct file_operations { loff_t (*llseek) (struct file *, loff_t, int); - ssize_t (*read) (struct file *, char *, size_t, loff_t *); - ssize_t (*write) (struct file *, const char *, size_t, loff_t *); + ssize_t (*read) (struct file *, char __user *, size_t, loff_t *); + ssize_t (*aio_read) (struct kiocb *, char __user *, size_t, loff_t); + ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); + ssize_t (*aio_write) (struct kiocb *, const char __user *, size_t, loff_t); int (*readdir) (struct file *, void *, filldir_t); unsigned int (*poll) (struct file *, struct poll_table_struct *); int (*ioctl) (struct inode *, struct file *, unsigned int, unsigned long); + long (*unlocked_ioctl) (struct file *, unsigned int, unsigned long); + long (*compat_ioctl) (struct file *, unsigned int, unsigned long); int (*mmap) (struct file *, struct vm_area_struct *); int (*open) (struct inode *, struct file *); + int (*flush) (struct file *); int (*release) (struct inode *, struct file *); - int (*fsync) (struct file *, struct dentry *); - int (*fasync) (struct file *, int); - int (*check_media_change) (kdev_t dev); - int (*revalidate) (kdev_t dev); + int (*fsync) (struct file *, struct dentry *, int datasync); + int (*aio_fsync) (struct kiocb *, int datasync); + int (*fasync) (int, struct file *, int); int (*lock) (struct file *, int, struct file_lock *); + ssize_t (*readv) (struct file *, const struct iovec *, unsigned long, loff_t *); + ssize_t (*writev) (struct file *, const struct iovec *, unsigned long, loff_t *); + ssize_t (*sendfile) (struct file *, loff_t *, size_t, read_actor_t, void *); + ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int); + unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long); + int (*check_flags)(int); + int (*dir_notify)(struct file *filp, unsigned long arg); + int (*flock) (struct file *, int, struct file_lock *); }; Again, all methods are called without any locks being held, unless @@ -362,8 +544,12 @@ otherwise noted. read: called by read(2) and related system calls + aio_read: called by io_submit(2) and other asynchronous I/O operations + write: called by write(2) and related system calls + aio_write: called by io_submit(2) and other asynchronous I/O operations + readdir: called when the VFS needs to read the directory contents poll: called by the VFS when a process wants to check if there is @@ -372,18 +558,25 @@ otherwise noted. ioctl: called by the ioctl(2) system call + unlocked_ioctl: called by the ioctl(2) system call. Filesystems that do not + require the BKL should use this method instead of the ioctl() above. + + compat_ioctl: called by the ioctl(2) system call when 32 bit system calls + are used on 64 bit kernels. + mmap: called by the mmap(2) system call open: called by the VFS when an inode should be opened. When the VFS - opens a file, it creates a new "struct file" and initialises - the "f_op" file operations member with the "default_file_ops" - field in the inode structure. It then calls the open method - for the newly allocated file structure. You might think that - the open method really belongs in "struct inode_operations", - and you may be right. I think it's done the way it is because - it makes filesystems simpler to implement. The open() method - is a good place to initialise the "private_data" member in the - file structure if you want to point to a device structure + opens a file, it creates a new "struct file". It then calls the + open method for the newly allocated file structure. You might + think that the open method really belongs in + "struct inode_operations", and you may be right. I think it's + done the way it is because it makes filesystems simpler to + implement. The open() method is a good place to initialize the + "private_data" member in the file structure if you want to point + to a device structure + + flush: called by the close(2) system call to flush a file release: called when the last reference to an open file is closed @@ -392,6 +585,23 @@ otherwise noted. fasync: called by the fcntl(2) system call when asynchronous (non-blocking) mode is enabled for a file + lock: called by the fcntl(2) system call for F_GETLK, F_SETLK, and F_SETLKW + commands + + readv: called by the readv(2) system call + + writev: called by the writev(2) system call + + sendfile: called by the sendfile(2) system call + + get_unmapped_area: called by the mmap(2) system call + + check_flags: called by the fcntl(2) system call for F_SETFL command + + dir_notify: called by the fcntl(2) system call for F_NOTIFY command + + flock: called by the flock(2) system call + Note that the file operations are implemented by the specific filesystem in which the inode resides. When opening a device node (character or block special) most filesystems will call special @@ -400,29 +610,28 @@ driver information. These support routines replace the filesystem file operations with those for the device driver, and then proceed to call the new open() method for the file. This is how opening a device file in the filesystem eventually ends up calling the device driver open() -method. Note the devfs (the Device FileSystem) has a more direct path -from device node to device driver (this is an unofficial kernel -patch). +method. -Directory Entry Cache (dcache) <section> ------------------------------- +Directory Entry Cache (dcache) +============================== + struct dentry_operations -======================== +------------------------ This describes how a filesystem can overload the standard dentry operations. Dentries and the dcache are the domain of the VFS and the individual filesystem implementations. Device drivers have no business here. These methods may be set to NULL, as they are either optional or -the VFS uses a default. As of kernel 2.1.99, the following members are +the VFS uses a default. As of kernel 2.6.13, the following members are defined: struct dentry_operations { - int (*d_revalidate)(struct dentry *); + int (*d_revalidate)(struct dentry *, struct nameidata *); int (*d_hash) (struct dentry *, struct qstr *); int (*d_compare) (struct dentry *, struct qstr *, struct qstr *); - void (*d_delete)(struct dentry *); + int (*d_delete)(struct dentry *); void (*d_release)(struct dentry *); void (*d_iput)(struct dentry *, struct inode *); }; @@ -451,6 +660,7 @@ Each dentry has a pointer to its parent dentry, as well as a hash list of child dentries. Child dentries are basically like files in a directory. + Directory Entry Cache APIs -------------------------- @@ -471,7 +681,7 @@ manipulate dentries: "d_delete" method is called d_drop: this unhashes a dentry from its parents hash list. A - subsequent call to dput() will dellocate the dentry if its + subsequent call to dput() will deallocate the dentry if its usage count drops to 0 d_delete: delete a dentry. If there are no other open references to @@ -507,16 +717,16 @@ up by walking the tree starting with the first component of the pathname and using that dentry along with the next component to look up the next level and so on. Since it is a frequent operation for workloads like multiuser -environments and webservers, it is important to optimize +environments and web servers, it is important to optimize this path. Prior to 2.5.10, dcache_lock was acquired in d_lookup and thus in every component during path look-up. Since 2.5.10 onwards, -fastwalk algorithm changed this by holding the dcache_lock +fast-walk algorithm changed this by holding the dcache_lock at the beginning and walking as many cached path component -dentries as possible. This signficantly decreases the number +dentries as possible. This significantly decreases the number of acquisition of dcache_lock. However it also increases the -lock hold time signficantly and affects performance in large +lock hold time significantly and affects performance in large SMP machines. Since 2.5.62 kernel, dcache has been using a new locking model that uses RCU to make dcache look-up lock-free. @@ -527,7 +737,7 @@ protected the hash chain, d_child, d_alias, d_lru lists as well as d_inode and several other things like mount look-up. RCU-based changes affect only the way the hash chain is protected. For everything else the dcache_lock must be taken for both traversing as well as -updating. The hash chain updations too take the dcache_lock. +updating. The hash chain updates too take the dcache_lock. The significant change is the way d_lookup traverses the hash chain, it doesn't acquire the dcache_lock for this and rely on RCU to ensure that the dentry has not been *freed*. @@ -535,14 +745,15 @@ ensure that the dentry has not been *freed*. Dcache locking details ---------------------- + For many multi-user workloads, open() and stat() on files are very frequently occurring operations. Both involve walking of path names to find the dentry corresponding to the concerned file. In 2.4 kernel, dcache_lock was held during look-up of each path component. Contention and -cacheline bouncing of this global lock caused significant +cache-line bouncing of this global lock caused significant scalability problems. With the introduction of RCU -in linux kernel, this was worked around by making +in Linux kernel, this was worked around by making the look-up of path components during path walking lock-free. @@ -562,7 +773,7 @@ Some of the important changes are : 2. Insertion of a dentry into the hash table is done using hlist_add_head_rcu() which take care of ordering the writes - the writes to the dentry must be visible before the dentry - is inserted. This works in conjuction with hlist_for_each_rcu() + is inserted. This works in conjunction with hlist_for_each_rcu() while walking the hash chain. The only requirement is that all initialization to the dentry must be done before hlist_add_head_rcu() since we don't have dcache_lock protection while traversing @@ -584,7 +795,7 @@ Some of the important changes are : the same. In some sense, dcache_rcu path walking looks like the pre-2.5.10 version. -5. All dentry hash chain updations must take the dcache_lock as well as +5. All dentry hash chain updates must take the dcache_lock as well as the per-dentry lock in that order. dput() does this to ensure that a dentry that has just been looked up in another CPU doesn't get deleted before dget() can be done on it. @@ -640,10 +851,10 @@ handled as described below : Since we redo the d_parent check and compare name while holding d_lock, lock-free look-up will not race against d_move(). -4. There can be a theoritical race when a dentry keeps coming back +4. There can be a theoretical race when a dentry keeps coming back to original bucket due to double moves. Due to this look-up may consider that it has never moved and can end up in a infinite loop. - But this is not any worse that theoritical livelocks we already + But this is not any worse that theoretical livelocks we already have in the kernel. |