If an incomplete character is passed to mbtowc(3), set errno to EILSEQ.

This is unambiguously required both by POSIX and by our own manual page.

It also makes a lot of sense because having a function that can
fail and that sets errno in some failure modes but does not set
errno in other failure modes would be a terrible idea.  Such a
function would be ridiculously complicated to use.  To detect the
reason for failure, you would have to:

 - save errno
 - reset errno to zero
 - call the function
 - inspect the return value to detect failure
 - inspect errno to decide about the reason for failure
 - if errno is zero, restore the saved errno

That is completely unreasonable, in particular for a seemingly innocous
function like mbtowc(3).  Next to no programmer would get that right in
any real-world program.

Note that this bug is very widespread, it also affects NetBSD,
FreeBSD, Solaris 11, and glibc.  I will also send a note around to
the other systems.

There may be fallout from programs using the interface incorrectly.
If you run into any, please report that to me.

OK millert@

1 files changed, 11 insertions, 4 deletions

diff --git a/lib/libc/locale/mbtowc.c b/lib/libc/locale/mbtowc.c
index 920f4bf26ed..4399ed8402f 100644
--- a/lib/libc/locale/mbtowc.c
+++ b/lib/libc/locale/mbtowc.c
@@ -1,4 +1,4 @@
-/*	$OpenBSD: mbtowc.c,v 1.2 2012/12/05 23:20:00 deraadt Exp $ */
+/*	$OpenBSD: mbtowc.c,v 1.3 2016/02/27 14:02:13 schwarze Exp $ */
 
 /*-
  * Copyright (c) 2002-2004 Tim J. Robbins.
@@ -44,7 +44,14 @@ mbtowc(wchar_t * __restrict pwc, const char * __restrict s, size_t n)
 		return (0);
 	}
 	rval = mbrtowc(pwc, s, n, &mbs);
-	if (rval == (size_t)-1 || rval == (size_t)-2)
-		return (-1);
-	return ((int)rval);
+
+	switch (rval) {
+	case (size_t)-2:
+		errno = EILSEQ;
+		/* FALLTHROUGH */
+	case (size_t)-1:
+		return -1;
+	default:
+		return (int)rval;
+	}
 }