diff options
author | dfr <dfr@FreeBSD.org> | 2008-03-26 15:23:12 +0000 |
---|---|---|
committer | dfr <dfr@FreeBSD.org> | 2008-03-26 15:23:12 +0000 |
commit | 79d2dfdaa69db38c43daed9744a6dbd0568189b5 (patch) | |
tree | eaf6a0fa52bc76253126814ddab4cbf78722a8a5 /usr.sbin | |
parent | 632e5d39f769e15274575347233e9d7aa364c0d6 (diff) | |
download | FreeBSD-src-79d2dfdaa69db38c43daed9744a6dbd0568189b5.zip FreeBSD-src-79d2dfdaa69db38c43daed9744a6dbd0568189b5.tar.gz |
Add the new kernel-mode NFS Lock Manager. To use it instead of the
user-mode lock manager, build a kernel with the NFSLOCKD option and
add '-k' to 'rpc_lockd_flags' in rc.conf.
Highlights include:
* Thread-safe kernel RPC client - many threads can use the same RPC
client handle safely with replies being de-multiplexed at the socket
upcall (typically driven directly by the NIC interrupt) and handed
off to whichever thread matches the reply. For UDP sockets, many RPC
clients can share the same socket. This allows the use of a single
privileged UDP port number to talk to an arbitrary number of remote
hosts.
* Single-threaded kernel RPC server. Adding support for multi-threaded
server would be relatively straightforward and would follow
approximately the Solaris KPI. A single thread should be sufficient
for the NLM since it should rarely block in normal operation.
* Kernel mode NLM server supporting cancel requests and granted
callbacks. I've tested the NLM server reasonably extensively - it
passes both my own tests and the NFS Connectathon locking tests
running on Solaris, Mac OS X and Ubuntu Linux.
* Userland NLM client supported. While the NLM server doesn't have
support for the local NFS client's locking needs, it does have to
field async replies and granted callbacks from remote NLMs that the
local client has contacted. We relay these replies to the userland
rpc.lockd over a local domain RPC socket.
* Robust deadlock detection for the local lock manager. In particular
it will detect deadlocks caused by a lock request that covers more
than one blocking request. As required by the NLM protocol, all
deadlock detection happens synchronously - a user is guaranteed that
if a lock request isn't rejected immediately, the lock will
eventually be granted. The old system allowed for a 'deferred
deadlock' condition where a blocked lock request could wake up and
find that some other deadlock-causing lock owner had beaten them to
the lock.
* Since both local and remote locks are managed by the same kernel
locking code, local and remote processes can safely use file locks
for mutual exclusion. Local processes have no fairness advantage
compared to remote processes when contending to lock a region that
has just been unlocked - the local lock manager enforces a strict
first-come first-served model for both local and remote lockers.
Sponsored by: Isilon Systems
PR: 95247 107555 115524 116679
MFC after: 2 weeks
Diffstat (limited to 'usr.sbin')
-rw-r--r-- | usr.sbin/Makefile | 1 | ||||
-rw-r--r-- | usr.sbin/clear_locks/Makefile | 8 | ||||
-rw-r--r-- | usr.sbin/clear_locks/clear_locks.8 | 51 | ||||
-rw-r--r-- | usr.sbin/clear_locks/clear_locks.c | 70 | ||||
-rw-r--r-- | usr.sbin/rpc.lockd/lockd.c | 268 | ||||
-rw-r--r-- | usr.sbin/rpc.lockd/rpc.lockd.8 | 6 |
6 files changed, 387 insertions, 17 deletions
diff --git a/usr.sbin/Makefile b/usr.sbin/Makefile index 4606f10..36094cf 100644 --- a/usr.sbin/Makefile +++ b/usr.sbin/Makefile @@ -32,6 +32,7 @@ SUBDIR= ac \ chown \ chroot \ ckdist \ + clear_locks \ config \ cron \ crunch \ diff --git a/usr.sbin/clear_locks/Makefile b/usr.sbin/clear_locks/Makefile new file mode 100644 index 0000000..8f28915 --- /dev/null +++ b/usr.sbin/clear_locks/Makefile @@ -0,0 +1,8 @@ +# $FreeBSD$ + +PROG= clear_locks +MAN= clear_locks.8 +LDADD= -lrpcsvc +WARNS= 6 + +.include <bsd.prog.mk> diff --git a/usr.sbin/clear_locks/clear_locks.8 b/usr.sbin/clear_locks/clear_locks.8 new file mode 100644 index 0000000..9f6cafe --- /dev/null +++ b/usr.sbin/clear_locks/clear_locks.8 @@ -0,0 +1,51 @@ +.\" Copyright (c) 2008 Isilon Inc http://www.isilon.com/ +.\" Authors: Doug Rabson <dfr@rabson.org> +.\" Developed with Red Inc: Alfred Perlstein <alfred@freebsd.org> +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd March 19, 2008 +.Dt CLEAR_LOCKS 8 +.Os +.Sh NAME +.Nm clear_locks +.Nd clear locks held on behalf of an NFS client +.Sh SYNOPSIS +.Nm +.Ar hostname +.Sh DESCRIPTION +The +.Nm +command can be used to clear file locks held by an NFS client. +This should only be used to handle problems caused by an NFS client +crashing while holding locks and failing to clear them itself when it +reboots. +.Sh SEE ALSO +.Xr rpc.lockd 8 +.Sh HISTORY +A version of +.Nm +appeared in +.Tn SunOS +4. diff --git a/usr.sbin/clear_locks/clear_locks.c b/usr.sbin/clear_locks/clear_locks.c new file mode 100644 index 0000000..1249c12 --- /dev/null +++ b/usr.sbin/clear_locks/clear_locks.c @@ -0,0 +1,70 @@ +/*- + * Copyright (c) 2008 Isilon Inc http://www.isilon.com/ + * Authors: Doug Rabson <dfr@rabson.org> + * Developed with Red Inc: Alfred Perlstein <alfred@freebsd.org> + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <sys/cdefs.h> +__FBSDID("$FreeBSD$"); + +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> + +#include <rpc/rpc.h> +#include <rpcsvc/nlm_prot.h> + +int +main(int argc, char **argv) +{ + enum clnt_stat stat; + char *hostname; + nlm4_notify notify; + + if (argc != 2) { + fprintf(stderr, "Usage: clear_locks <hostname>\n"); + exit(1); + } + hostname = argv[1]; + + if (geteuid() != 0) { + fprintf(stderr, "clear_locks: must be root\n"); + exit(1); + } + + notify.name = hostname; + notify.state = 0; + stat = rpc_call("localhost", NLM_PROG, NLM_VERS4, NLM4_FREE_ALL, + (xdrproc_t) xdr_nlm4_notify, (void *) ¬ify, + (xdrproc_t) xdr_void, NULL, NULL); + + if (stat != RPC_SUCCESS) { + clnt_perrno(stat); + exit(1); + } + fprintf(stderr, "clear_locks: cleared locks for hostname %s\n", + hostname); + + return (0); +} diff --git a/usr.sbin/rpc.lockd/lockd.c b/usr.sbin/rpc.lockd/lockd.c index 99081d4..3b1572e 100644 --- a/usr.sbin/rpc.lockd/lockd.c +++ b/usr.sbin/rpc.lockd/lockd.c @@ -48,6 +48,7 @@ __RCSID("$NetBSD: lockd.c,v 1.7 2000/08/12 18:08:44 thorpej Exp $"); #include <sys/types.h> #include <sys/socket.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> @@ -76,13 +77,17 @@ int _rpcsvcdirty = 0; int grace_expired; int nsm_state; +int kernel_lockd; pid_t client_pid; struct mon mon_host; char **hosts, *svcport_str = NULL; int nhosts = 0; int xcreated = 0; +char **addrs; /* actually (netid, uaddr) pairs */ +int naddrs; /* count of how many (netid, uaddr) pairs */ void create_service(struct netconfig *nconf); +void lookup_addresses(struct netconfig *nconf); void init_nsm(void); void nlm_prog_0(struct svc_req *, SVCXPRT *); void nlm_prog_1(struct svc_req *, SVCXPRT *); @@ -93,6 +98,11 @@ void usage(void); void sigalarm_handler(void); +/* + * XXX move to some header file. + */ +#define _PATH_RPCLOCKDSOCK "/var/run/rpclockd.sock" + int main(int argc, char **argv) { @@ -106,7 +116,7 @@ main(int argc, char **argv) int maxrec = RPC_MAXDATASIZE; in_port_t svcport = 0; - while ((ch = getopt(argc, argv, "d:g:h:p:")) != (-1)) { + while ((ch = getopt(argc, argv, "d:g:h:kp:")) != (-1)) { switch (ch) { case 'd': debug_level = atoi(optarg); @@ -143,6 +153,9 @@ main(int argc, char **argv) out_of_mem(); } break; + case 'k': + kernel_lockd = TRUE; + break; case 'p': endptr = NULL; svcport = (in_port_t)strtoul(optarg, &endptr, 10); @@ -221,19 +234,77 @@ main(int argc, char **argv) hosts[nhosts - 1] = "127.0.0.1"; } - nc_handle = setnetconfig(); - while ((nconf = getnetconfig(nc_handle))) { - /* We want to listen only on udp6, tcp6, udp, tcp transports */ - if (nconf->nc_flag & NC_VISIBLE) { - /* Skip if there's no IPv6 support */ - if (have_v6 == 0 && strcmp(nconf->nc_protofmly, "inet6") == 0) { - /* DO NOTHING */ - } else { - create_service(nconf); + if (kernel_lockd) { + /* + * For the kernel lockd case, we run a cut-down RPC + * service on a local-domain socket. The kernel's RPC + * server will pass what it can't handle (mainly + * client replies) down to us. This can go away + * entirely if/when we move the client side of NFS + * locking into the kernel. + */ + struct sockaddr_un sun; + int fd, oldmask; + SVCXPRT *xprt; + + memset(&sun, 0, sizeof sun); + sun.sun_family = AF_LOCAL; + unlink(_PATH_RPCLOCKDSOCK); + strcpy(sun.sun_path, _PATH_RPCLOCKDSOCK); + sun.sun_len = SUN_LEN(&sun); + fd = socket(AF_LOCAL, SOCK_STREAM, 0); + if (!fd) { + err(1, "Can't create local lockd socket"); + } + oldmask = umask(S_IXUSR|S_IRWXG|S_IRWXO); + if (bind(fd, (struct sockaddr *) &sun, sun.sun_len) < 0) { + err(1, "Can't bind local lockd socket"); + } + umask(oldmask); + if (listen(fd, SOMAXCONN) < 0) { + err(1, "Can't listen on local lockd socket"); + } + xprt = svc_vc_create(fd, RPC_MAXDATASIZE, RPC_MAXDATASIZE); + if (!xprt) { + err(1, "Can't create transport for local lockd socket"); + } + if (!svc_reg(xprt, NLM_PROG, NLM_VERS4, nlm_prog_4, NULL)) { + err(1, "Can't register service for local lockd socket"); + } + + /* + * We need to look up the addresses so that we can + * hand uaddrs (ascii encoded address+port strings) to + * the kernel. + */ + nc_handle = setnetconfig(); + while ((nconf = getnetconfig(nc_handle))) { + /* We want to listen only on udp6, tcp6, udp, tcp transports */ + if (nconf->nc_flag & NC_VISIBLE) { + /* Skip if there's no IPv6 support */ + if (have_v6 == 0 && strcmp(nconf->nc_protofmly, "inet6") == 0) { + /* DO NOTHING */ + } else { + lookup_addresses(nconf); + } + } + } + endnetconfig(nc_handle); + } else { + nc_handle = setnetconfig(); + while ((nconf = getnetconfig(nc_handle))) { + /* We want to listen only on udp6, tcp6, udp, tcp transports */ + if (nconf->nc_flag & NC_VISIBLE) { + /* Skip if there's no IPv6 support */ + if (have_v6 == 0 && strcmp(nconf->nc_protofmly, "inet6") == 0) { + /* DO NOTHING */ + } else { + create_service(nconf); + } } } + endnetconfig(nc_handle); } - endnetconfig(nc_handle); /* * Note that it is NOT sensible to run this program from inetd - the @@ -259,14 +330,28 @@ main(int argc, char **argv) strerror(errno)); exit(1); } - grace_expired = 0; - alarm(grace_period); - init_nsm(); + if (kernel_lockd) { + client_pid = client_request(); + + /* + * Create a child process to enter the kernel and then + * wait for RPCs on our local domain socket. + */ + if (!fork()) + nlm_syscall(debug_level, grace_period, naddrs, addrs); + else + svc_run(); + } else { + grace_expired = 0; + alarm(grace_period); + + init_nsm(); - client_pid = client_request(); + client_pid = client_request(); - svc_run(); /* Should never return */ + svc_run(); /* Should never return */ + } exit(1); } @@ -499,6 +584,155 @@ create_service(struct netconfig *nconf) } /* end while */ } +/* + * Look up addresses for the kernel to create transports for. + */ +void +lookup_addresses(struct netconfig *nconf) +{ + struct addrinfo hints, *res = NULL; + struct sockaddr_in *sin; + struct sockaddr_in6 *sin6; + struct __rpc_sockinfo si; + struct netbuf servaddr; + SVCXPRT *transp = NULL; + int aicode; + int nhostsbak; + int r; + int registered = 0; + u_int32_t host_addr[4]; /* IPv4 or IPv6 */ + char *uaddr; + + if ((nconf->nc_semantics != NC_TPI_CLTS) && + (nconf->nc_semantics != NC_TPI_COTS) && + (nconf->nc_semantics != NC_TPI_COTS_ORD)) + return; /* not my type */ + + /* + * XXX - using RPC library internal functions. + */ + if (!__rpc_nconf2sockinfo(nconf, &si)) { + syslog(LOG_ERR, "cannot get information for %s", + nconf->nc_netid); + return; + } + + /* Get rpc.statd's address on this transport */ + memset(&hints, 0, sizeof hints); + hints.ai_flags = AI_PASSIVE; + hints.ai_family = si.si_af; + hints.ai_socktype = si.si_socktype; + hints.ai_protocol = si.si_proto; + + /* + * Bind to specific IPs if asked to + */ + nhostsbak = nhosts; + while (nhostsbak > 0) { + --nhostsbak; + + switch (hints.ai_family) { + case AF_INET: + if (inet_pton(AF_INET, hosts[nhostsbak], + host_addr) == 1) { + hints.ai_flags &= AI_NUMERICHOST; + } else { + /* + * Skip if we have an AF_INET6 address. + */ + if (inet_pton(AF_INET6, hosts[nhostsbak], + host_addr) == 1) { + continue; + } + } + break; + case AF_INET6: + if (inet_pton(AF_INET6, hosts[nhostsbak], + host_addr) == 1) { + hints.ai_flags &= AI_NUMERICHOST; + } else { + /* + * Skip if we have an AF_INET address. + */ + if (inet_pton(AF_INET, hosts[nhostsbak], + host_addr) == 1) { + continue; + } + } + break; + default: + break; + } + + /* + * If no hosts were specified, just bind to INADDR_ANY + */ + if (strcmp("*", hosts[nhostsbak]) == 0) { + if (svcport_str == NULL) { + res = malloc(sizeof(struct addrinfo)); + if (res == NULL) + out_of_mem(); + res->ai_flags = hints.ai_flags; + res->ai_family = hints.ai_family; + res->ai_protocol = hints.ai_protocol; + switch (res->ai_family) { + case AF_INET: + sin = malloc(sizeof(struct sockaddr_in)); + if (sin == NULL) + out_of_mem(); + sin->sin_family = AF_INET; + sin->sin_port = htons(0); + sin->sin_addr.s_addr = htonl(INADDR_ANY); + res->ai_addr = (struct sockaddr*) sin; + res->ai_addrlen = (socklen_t) + sizeof(res->ai_addr); + break; + case AF_INET6: + sin6 = malloc(sizeof(struct sockaddr_in6)); + if (sin6 == NULL) + out_of_mem(); + sin6->sin6_family = AF_INET6; + sin6->sin6_port = htons(0); + sin6->sin6_addr = in6addr_any; + res->ai_addr = (struct sockaddr*) sin6; + res->ai_addrlen = (socklen_t) sizeof(res->ai_addr); + break; + default: + break; + } + } else { + if ((aicode = getaddrinfo(NULL, svcport_str, + &hints, &res)) != 0) { + syslog(LOG_ERR, + "cannot get local address for %s: %s", + nconf->nc_netid, + gai_strerror(aicode)); + continue; + } + } + } else { + if ((aicode = getaddrinfo(hosts[nhostsbak], svcport_str, + &hints, &res)) != 0) { + syslog(LOG_ERR, + "cannot get local address for %s: %s", + nconf->nc_netid, gai_strerror(aicode)); + continue; + } + } + + servaddr.len = servaddr.maxlen = res->ai_addr->sa_len; + servaddr.buf = res->ai_addr; + uaddr = taddr2uaddr(nconf, &servaddr); + + addrs = realloc(addrs, 2 * (naddrs + 1) * sizeof(char *)); + if (!addrs) + out_of_mem(); + addrs[2 * naddrs] = strdup(nconf->nc_netid); + addrs[2 * naddrs + 1] = uaddr; + naddrs++; + } /* end while */ +} + void sigalarm_handler(void) { @@ -509,7 +743,7 @@ sigalarm_handler(void) void usage() { - errx(1, "usage: rpc.lockd [-d <debuglevel>]" + errx(1, "usage: rpc.lockd [-k] [-d <debuglevel>]" " [-g <grace period>] [-h <bindip>] [-p <port>]"); } diff --git a/usr.sbin/rpc.lockd/rpc.lockd.8 b/usr.sbin/rpc.lockd/rpc.lockd.8 index 4fabe5d..9cd429b 100644 --- a/usr.sbin/rpc.lockd/rpc.lockd.8 +++ b/usr.sbin/rpc.lockd/rpc.lockd.8 @@ -41,6 +41,7 @@ .Nd NFS file locking daemon .Sh SYNOPSIS .Nm +.Op Fl k .Op Fl d Ar debug_level .Op Fl g Ar grace period .Op Fl h Ar bindip @@ -58,6 +59,11 @@ with Options and operands available for .Nm : .Bl -tag -width indent +.It Fl k +The +.Fl k +option specifies the use of the kernel-resident NFS lock manager, if +possible. .It Fl d The .Fl d |