Several improvements to escape sequence handling.

* Add the missing special character \_ (underscore).
* Partial implementations of \a (leader character)
and \E (uninterpreted escape character).
* Parse and ignore \r (reverse line feed).
* Add a WARNING message about undefined escape sequences.
* Add an UNSUPP message about unsupported escape sequences.
* Mark \! and \? (transparent throughput)
and \O (suppress output) as unsupported.
* Treat the various variants of zero-width spaces as one-byte escape
sequences rather than as special characters, to avoid defining bogus
forms with square brackets.
* For special characters with one-byte names, do not define bogus
forms with square brackets, except for \[-], which is valid.
* In the form with square brackets, undefined special characters do not
fall back to printing the name verbatim, not even for one-byte names.
* Starting a special character name with a blank is an error.
* Undefined escape sequences never abort formatting of the input
string, not even in HTML output mode.
* Document the newly handled escapes, and a few that were missing.
* Regression tests for most of the above.

2 files changed, 52 insertions, 7 deletions

diff --git a/share/man/man7/mandoc_char.7 b/share/man/man7/mandoc_char.7
index a85c02848b7..d6c905beab3 100644
--- a/share/man/man7/mandoc_char.7
+++ b/share/man/man7/mandoc_char.7
@@ -1,8 +1,8 @@
-.\"	$OpenBSD: mandoc_char.7,v 1.39 2018/08/21 01:56:26 schwarze Exp $
+.\"	$OpenBSD: mandoc_char.7,v 1.40 2018/12/15 19:30:19 schwarze Exp $
 .\"
 .\" Copyright (c) 2003 Jason McIntyre <jmc@openbsd.org>
 .\" Copyright (c) 2009, 2010, 2011 Kristaps Dzonsons <kristaps@bsd.lv>
-.\" Copyright (c) 2011, 2013, 2015, 2017 Ingo Schwarze <schwarze@openbsd.org>
+.\" Copyright (c) 2011,2013,2015,2017,2018 Ingo Schwarze <schwarze@openbsd.org>
 .\"
 .\" Permission to use, copy, modify, and distribute this software for any
 .\" purpose with or without fee is hereby granted, provided that the above
@@ -16,7 +16,7 @@
 .\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 .\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 .\"
-.Dd $Mdocdate: August 21 2018 $
+.Dd $Mdocdate: December 15 2018 $
 .Dt MANDOC_CHAR 7
 .Os
 .Sh NAME
@@ -266,11 +266,13 @@ Spacing:
 .It Em Input Ta Em Description
 .It Sq \e\ \& Ta unpaddable non-breaking space
 .It \e\(ti   Ta paddable non-breaking space
-.It \e0      Ta unpaddable, breaking digit-width space
+.It \e0      Ta digit-width space allowing line break
 .It \e|      Ta one-sixth \e(em narrow space, zero width in nroff mode
 .It \e^      Ta one-twelfth \e(em half-narrow space, zero width in nroff
-.It \e&      Ta zero-width space
+.It \e&      Ta zero-width non-breaking space
+.It \e)      Ta zero-width space transparent to end-of-sentence detection
 .It \e%      Ta zero-width space allowing hyphenation
+.It \e:      Ta zero-width space allowing line break
 .El
 .Pp
 Lines:
diff --git a/share/man/man7/roff.7 b/share/man/man7/roff.7
index 06531d1e7db..13bc85e2632 100644
--- a/share/man/man7/roff.7
+++ b/share/man/man7/roff.7
@@ -1,4 +1,4 @@
-.\"	$OpenBSD: roff.7,v 1.87 2018/10/04 15:31:40 schwarze Exp $
+.\"	$OpenBSD: roff.7,v 1.88 2018/12/15 19:30:19 schwarze Exp $
 .\"
 .\" Copyright (c) 2010, 2011, 2012 Kristaps Dzonsons <kristaps@bsd.lv>
 .\" Copyright (c) 2010-2018 Ingo Schwarze <schwarze@openbsd.org>
@@ -15,7 +15,7 @@
 .\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 .\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 .\"
-.Dd $Mdocdate: October 4 2018 $
+.Dd $Mdocdate: December 15 2018 $
 .Dt ROFF 7
 .Os
 .Sh NAME
@@ -1842,6 +1842,11 @@ The escape sequence backslash-space
 .Pq Sq \e\ \&
 is an unpaddable space-sized non-breaking space character; see
 .Sx Whitespace .
+.It Ic \e!
+Embed text up to and including the end of the input line into the
+current diversion or into intermediate output without interpreting
+requests, macros, and escapes.
+Currently unsupported.
 .It Ic \e\(dq
 The rest of the input line is treated as
 .Sx Comments .
@@ -1868,6 +1873,10 @@ instead.
 .Sx Special Characters
 with two-letter names, see
 .Xr mandoc_char 7 .
+.It Ic \e)
+Zero-width space transparent to end-of-sentence detection;
+ignored by
+.Xr mandoc 1 .
 .It Ic \e*[ Ns Ar name Ns Ic \&]
 Interpolate the string with the
 .Ar name .
@@ -1905,6 +1914,15 @@ Special character
 .It Ic \e/
 Right italic correction (groff extension); ignored by
 .Xr mandoc 1 .
+.It Ic \e:
+Breaking the line is allowed at this point of the word
+without inserting a hyphen.
+.It Ic \e?
+Embed the text up to the next
+.Ic \e?
+into the current diversion without interpreting requests, macros,
+and escapes.
+This is a groff extension and currently unsupported.
 .It Ic \e[ Ns Ar name Ns Ic \&]
 .Sx Special Characters
 with names of arbitrary length, see
@@ -1912,6 +1930,10 @@ with names of arbitrary length, see
 .It Ic \e^
 One-twelfth em half-narrow space character, effectively zero-width in
 .Xr mandoc 1 .
+.It Ic \e_
+Underline special character; use
+.Ic \e(ul
+instead.
 .It Ic \e`
 Grave accent special character; use
 .Ic \e(ga
@@ -1932,6 +1954,9 @@ Digit width space character.
 .It Ic \eA\(aq Ns Ar string Ns Ic \(aq
 Anchor definition; ignored by
 .Xr mandoc 1 .
+.It Ic \ea
+Leader character; ignored by
+.Xr mandoc 1 .
 .It Ic \eB\(aq Ns Ar string Ns Ic \(aq
 Interpolate
 .Sq 1
@@ -1959,6 +1984,13 @@ Draw graphics function; ignored by
 .It Ic \ed
 Move down by half a line; ignored by
 .Xr mandoc 1 .
+.It Ic \eE
+Escape character intended to not be interpreted in copy mode.
+In
+.Xr mandoc 1 ,
+it does the same as
+.Ic \e
+itself for now.
 .It Ic \ee
 Backslash special character.
 .It Ic \eF[ Ns Ar name Ns Ic \&]
@@ -2040,6 +2072,14 @@ the register is first incremented or decremented by the
 that was specified in the relevant
 .Ic \&nr
 request, and the changed value is interpolated.
+.It Ic \eO Ns Ar digit , Ic \eO[5 Ns arguments Ns Ic \&]
+Suppress output.
+This is a groff extension and currently unsupported.
+With an argument of
+.Ic 1 , 2 , 3 ,
+or
+.Ic 4 ,
+it is ignored.
 .It Ic \eo\(aq Ns Ar string Ns Ic \(aq
 Overstrike, writing all the characters contained in the
 .Ar string
@@ -2051,6 +2091,9 @@ Break the output line at the end of the current word.
 .It Ic \eR\(aq Ns Ar name Oo +|- Oc Ns Ar number Ns Ic \(aq
 Set number register; ignored by
 .Xr mandoc 1 .
+.It Ic \er
+Move up by one line; ignored by
+.Xr mandoc 1 .
 .It Ic \eS\(aq Ns Ar number Ns Ic \(aq
 Slant output; ignored by
 .Xr mandoc 1 .