[PATCH] pty_chars_in_buffer oops fix

The idea of this patch is to lock both sides of a ptmx/pty pair during line
discipline changing.  This is needed to ensure that say a poll on one side of
the pty doesn't occur while the line discipline is actively being changed.
This resulted in an oops reported on lkml, see:

	http://marc.theaimsgroup.com/?l=linux-kernel&m=111342171410005&w=2

A 'hacky' approach was previously implmemented which served to eliminate the
poll vs.  line discipline changing race.  However, this patch takes a more
general approach to the issue.  The patch only adds locking on a less often
used path, the line-discipline changing path, as opposed to locking the
ptmx/pty pair on read/write/poll paths.

The patch below, takes both ldisc locks in either order b/c the locks are both
taken under the same spinlock().  I thought about locking the ptmx/pty
separately, such as master always first but that introduces a 3 way deadlock.
For example, process 1 does a blocking read on the slave side.  Then, process
2 does an ldisc change on the slave side, which acquires the master ldisc lock
but not the slave's.  Finally, process 3 does a write which blocks on the
process 2's ldisc reference.

This patch does introduce some changes in semantics.  For example, a line
discipline change on side 'a' of a ptmx/pty pair, will now wait for a
read/write to complete on the other side, or side 'b'.  The current behavior
is to simply wait for any read/writes on only side 'a', not both sides 'a' and
'b'.  I think this behavior makes sense, but I wanted to point it out.

I've tested the patch with a bunch of read/write/poll while changing the line
discipline out from underneath.

This patch obviates the need for the above "hide the problem" patch.

Signed-off-by: Jason Baron <jbaron@redhat.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

1 files changed, 2 insertions, 3 deletions

diff --git a/drivers/char/pty.c b/drivers/char/pty.c
index da32889..49f3997 100644
--- a/drivers/char/pty.c
+++ b/drivers/char/pty.c
@@ -149,15 +149,14 @@ static int pty_write_room(struct tty_struct *tty)
 static int pty_chars_in_buffer(struct tty_struct *tty)
 {
 	struct tty_struct *to = tty->link;
-	ssize_t (*chars_in_buffer)(struct tty_struct *);
 	int count;
 
 	/* We should get the line discipline lock for "tty->link" */
-	if (!to || !(chars_in_buffer = to->ldisc.chars_in_buffer))
+	if (!to || !to->ldisc.chars_in_buffer)
 		return 0;
 
 	/* The ldisc must report 0 if no characters available to be read */
-	count = chars_in_buffer(to);
+	count = to->ldisc.chars_in_buffer(to);
 
 	if (tty->driver->subtype == PTY_TYPE_SLAVE) return count;