Add the BSD-licensed Citrus iconv to the base system with default off

setting. It can be built by setting the WITH_ICONV knob. While this
knob is unset, the library part, the binaries, the header file and
the metadata files will not be built or installed so it makes no impact
on the system if left turned off.

This work is based on the iconv implementation in NetBSD but a great
number of improvements and feature additions have been included:

- Some utilities have been added. There is a conversion table generator,
  which can compare conversion tables to reference data generated by
  GNU libiconv. This helps ensuring conversion compatibility.
- UTF-16 surrogate support and some endianness issues have been fixed.
- The rather chaotic Makefiles to build metadata have been refactored
  and cleaned up, now it is easy to read and it is also easier to add
  support for new encodings.
- A bunch of new encodings and encoding aliases have been added.
- Support for 1->2, 1->3 and 1->4 mappings, which is needed for
  transliterating with flying accents as GNU does, like "u.
- Lots of warnings have been fixed, the major part of the code is
  now WARNS=6 clean.
- New section 1 and section 5 manual pages have been added.
- Some GNU-specific calls have been implemented:
  iconvlist(), iconvctl(), iconv_canonicalize(), iconv_open_into()
- Support for GNU's //IGNORE suffix has been added.
- The "-" argument for stdin is now recognized in iconv(1) as per POSIX.
- The Big5 conversion module has been fixed.
- The iconv.h header files is supposed to be compatible with the
  GNU version, i.e. sources should build with base iconv.h and
  GNU libiconv. It also includes a macro magic to deal with the
  char ** and const char ** incompatibility.
- GNU compatibility: "" or "char" means the current local
  encoding in use
- Various cleanups and style(9) fixes.

Approved by:	delphij (mentor)
Obtained from:	The NetBSD Project
Sponsored by:	Google Summer of Code 2009

1 files changed, 426 insertions, 0 deletions

diff --git a/share/i18n/csmapper/APPLE/UCS%ARABIC.src b/share/i18n/csmapper/APPLE/UCS%ARABIC.src
new file mode 100644
index 0000000..bfcad10
--- /dev/null
+++ b/share/i18n/csmapper/APPLE/UCS%ARABIC.src
@@ -0,0 +1,426 @@
+# $FreeBSD$
+
+TYPE		ROWCOL
+NAME		UCS/ARABIC
+SRC_ZONE	0x0000-0xFB02
+OOB_MODE	INVALID
+DST_INVALID	0x100
+DST_UNIT_BITS	16
+
+BEGIN_MAP
+#=======================================================================
+#   File name:  ARABIC.TXT
+#
+#   Contents:   Map (external version) from Mac OS Arabic
+#               character set to Unicode 2.1 and later.
+#
+#   Copyright:  (c) 1994-2002, 2005 by Apple Computer, Inc., all rights
+#               reserved.
+#
+#   Contact:    charsets@apple.com
+#
+#   Changes:
+#
+#       c02  2005-Apr-04    Update header comments. Matches internal xml
+#                           <c1.2> and Text Encoding Converter 2.0.
+#      b3,c1 2002-Dec-19    Add comments about character display and
+#                           direction overrides. Update URLs, notes.
+#                           Matches internal utom<b4>.
+#       b02  1999-Sep-22    Update contact e-mail address. Matches
+#                           internal utom<b1>, ufrm<b1>, and Text
+#                           Encoding Converter version 1.5.
+#       n10  1998-Feb-05    Show required Unicode character
+#                           directionality in a different way. Matches
+#                           internal utom<n4>, ufrm<n21>, and Text
+#                           Encoding Converter version 1.3. Update
+#                           header comments; include information on
+#                           loose mapping of digits.
+#       n07  1997-Jul-17    Update to match internal utom<n2>, ufrm<n17>:
+#                           Change standard mapping for 0xC0 from U+066D
+#                           to U+274A. Add direction overrides to
+#                           mappings for 0x25, 0x2C, 0x3B, 0x3F. Add
+#                           information on variants.
+#       n03  1995-Apr-18    First version (after fixing some typos).
+#                           Matches internal ufrm<n11>.
+#
+# Standard header:
+# ----------------
+#
+#   Apple, the Apple logo, and Macintosh are trademarks of Apple
+#   Computer, Inc., registered in the United States and other countries.
+#   Unicode is a trademark of Unicode Inc. For the sake of brevity,
+#   throughout this document, "Macintosh" can be used to refer to
+#   Macintosh computers and "Unicode" can be used to refer to the
+#   Unicode standard.
+#
+#   Apple Computer, Inc. ("Apple") makes no warranty or representation,
+#   either express or implied, with respect to this document and the
+#   included data, its quality, accuracy, or fitness for a particular
+#   purpose. In no event will Apple be liable for direct, indirect, 
+#   special, incidental, or consequential damages resulting from any
+#   defect or inaccuracy in this document or the included data.
+#
+#   These mapping tables and character lists are subject to change.
+#   The latest tables should be available from the following:
+#
+#   <http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/>
+#
+#   For general information about Mac OS encodings and these mapping
+#   tables, see the file "README.TXT".
+#
+# Format:
+# -------
+#
+#   Three tab-separated columns;
+#   '#' begins a comment which continues to the end of the line.
+#     Column #1 is the Mac OS Arabic code (in hex as 0xNN).
+#     Column #2 is the corresponding Unicode (in hex as 0xNNNN),
+#       possibly preceded by a tag indicating required directionality
+#       (i.e. <LR>+0xNNNN or <RL>+0xNNNN).
+#     Column #3 is a comment containing the Unicode name.
+#
+#   The entries are in Mac OS Arabic code order.
+#
+#   Control character mappings are not shown in this table, following
+#   the conventions of the standard UTC mapping tables. However, the
+#   Mac OS Arabic character set uses the standard control characters at
+#   0x00-0x1F and 0x7F.
+#
+# Notes on Mac OS Arabic:
+# -----------------------
+#
+#   This is a legacy Mac OS encoding; in the Mac OS X Carbon and Cocoa
+#   environments, it is only supported via transcoding to and from
+#   Unicode.
+#
+#   1. General
+#
+#   The Mac OS Arabic character set is intended to cover Arabic as
+#   used in North Africa, the Arabian peninsula, and the Levant. It
+#   also contains several characters needed for Urdu and/or Farsi.
+#
+#   The Mac OS Arabic character set is essentially a superset of ISO
+#   8859-6. The 8859-6 code points that are interpreted differently
+#   in the Mac OS Arabic set are as follows:
+#    0xA0 is NO-BREAK SPACE in 8859-6 and right-left SPACE in Mac OS
+#         Arabic; NO-BREAK is 0x81 in Mac OS Arabic.
+#    0xA4 is CURRENCY SIGN in 8859-6 and right-left DOLLAR SIGN in
+#         Mac OS Arabic.
+#    0xAD is SOFT HYPHEN in 8859-6 and right-left HYPHEN-MINUS in
+#         Mac OS Arabic.
+#   ISO 8859-6 specifies that codes 0x30-0x39 can be rendered either
+#   with European digit shapes or Arabic digit shapes. This is also
+#   true in Mac OS Arabic, which determines from context which digit
+#   shapes to use (see below).
+#
+#   The Mac OS Arabic character set uses the C1 controls area and other
+#   code points which are undefined in ISO 8859-6 for additional
+#   graphic characters: additional Arabic letters for Farsi and Urdu,
+#   some accented Roman letters for European languages (such as French),
+#   and duplicates of some of the punctuation, symbols, and digits in
+#   the ASCII block. The duplicate punctuation, symbol, and digit
+#   characters have right-left directionality, while the ASCII versions
+#   have left-right directionality. See the next section for more
+#   information on this.
+#
+#   Mac OS Arabic characters 0xEB-0xF2 are non-spacing/combining marks.
+#
+#   2. Directional characters and roundtrip fidelity
+#
+#   The Mac OS Arabic character set was developed in 1986-1987. At that
+#   time the bidirectional line layout algorithm used in the Mac OS
+#   Arabic system was fairly simple; it used only a few direction
+#   classes (instead of the 19 now used in the Unicode bidirectional
+#   algorithm). In order to permit users to handle some tricky layout
+#   problems, certain punctuation and symbol characters were encoded
+#   twice, one with a left-right direction attribute and the other with
+#   a right-left direction attribute.
+#
+#   For example, plus sign is encoded at 0x2B with a left-right
+#   attribute, and at 0xAB with a right-left attribute. However, there
+#   is only one PLUS SIGN character in Unicode. This leads to some
+#   interesting problems when mapping between Mac OS Arabic and Unicode;
+#   see below.
+#
+#   A related problem is that even when a particular character is
+#   encoded only once in Mac OS Arabic, it may have a different
+#   direction attribute than the corresponding Unicode character.
+#
+#   For example, the Mac OS Arabic character at 0x93 is HORIZONTAL
+#   ELLIPSIS with strong right-left direction. However, the Unicode
+#   character HORIZONTAL ELLIPSIS has direction class neutral.
+#
+#   3. Behavior of ASCII-range numbers in WorldScript
+#
+#   Mac OS Arabic also has two sets of digit codes.
+#
+#   The digits at 0x30-0x39 may be displayed using either European
+#   digit forms or Arabic digit forms, depending on context. If there
+#   is a "strong European" character such as a Latin letter on either
+#   side of a sequence consisting of digits 0x30-0x39 and possibly comma
+#   0x2C or period 0x2E, then the characters will be displayed using
+#   European forms (This will happen even if there are neutral characters
+#   between the digits and the strong European character). Otherwise, the
+#   digits will be displayed using Arabic forms, the comma will be
+#   displayed as Arabic thousands separator, and the period as Arabic
+#   decimal separator. In any case, 0x2C, 0x2E, and 0x30-0x39 are always
+#   left-right.
+#
+#   The digits at 0xB0-0xB9 are always displayed using Arabic digit
+#   shapes, and moreover, these digits always have strong right-left
+#   directionality. These are mainly intended for special layout
+#   purposes such as part numbers, etc.
+#
+#   4. Font variants
+#
+#   The table in this file gives the Unicode mappings for the standard
+#   Mac OS Arabic encoding. This encoding is supported by the Cairo font
+#   (the system font for Arabic), and is the encoding supported by the
+#   text processing utilities. However, the other Arabic fonts actually
+#   implement slightly different encodings; this mainly affects the code
+#   points 0xAA and 0xC0. For these code points the standard Mac OS
+#   Arabic encoding has the following mappings:
+#     0xAA -> <RL>+0x002A ASTERISK, right-left
+#     0xC0 -> <RL>+0x274A EIGHT TEARDROP-SPOKED PROPELLER ASTERISK,
+#                         right-left
+#   This mapping of 0xAA is consistent with the normal convention for
+#   Mac OS Arabic and Hebrew that the right-left duplicates have codes
+#   that are equal to the ASCII code of the left-right character plus
+#   0x80. However, in all of the other fonts, 0xAA is MULTIPLY SIGN, and
+#   right-left ASTERISK may be at a different code point. The other
+#   variants are described below.
+#
+#   The TrueType variant is used for most of the Arabic TrueType fonts:
+#   Baghdad, Geeza, Kufi, Nadeem.  It differs from the standard variant
+#   in the following way:
+#     0xAA -> <RL>+0x00D7 MULTIPLICATION SIGN, right-left
+#     0xC0 -> <RL>+0x002A ASTERISK, right-left
+#
+#   The Thuluth variant is used for the Arabic Postscript-only fonts:
+#   Thuluth and Thuluth bold. It differs from the standard variant in
+#   the following way:
+#     0xAA -> <RL>+0x00D7 MULTIPLICATION SIGN, right-left
+#     0xC0 -> 0x066D ARABIC FIVE POINTED STAR
+#
+#   The AlBayan variant is used for the Arabic TrueType font Al Bayan.
+#   It differs from the standard variant in the following way:
+#     0x81 -> no mapping (glyph just has authorship information, etc.)
+#     0xA3 -> 0xFDFA ARABIC LIGATURE SALLALLAHOU ALAYHE WASALLAM
+#     0xA4 -> 0xFDF2 ARABIC LIGATURE ALLAH ISOLATED FORM
+#     0xAA -> <RL>+0x00D7 MULTIPLICATION SIGN, right-left
+#     0xDC -> <RL>+0x25CF BLACK CIRCLE, right-left
+#     0xFC -> <RL>+0x25A0 BLACK SQUARE, right-left
+#
+# Unicode mapping issues and notes:
+# ---------------------------------
+#
+#   1. Matching the direction of Mac OS Arabic characters
+#
+#   When Mac OS Arabic encodes a character twice but with different
+#   direction attributes for the two code points - as in the case of
+#   plus sign mentioned above - we need a way to map both Mac OS Arabic
+#   code points to Unicode and back again without loss of information.
+#   With the plus sign, for example, mapping one of the Mac OS Arabic
+#   characters to a code in the Unicode corporate use zone is
+#   undesirable, since both of the plus sign characters are likely to
+#   be used in text that is interchanged.
+#
+#   The problem is solved with the use of direction override characters
+#   and direction-dependent mappings. When mapping from Mac OS Arabic
+#   to Unicode, we use direction overrides as necessary to force the
+#   direction of the resulting Unicode characters.
+#
+#   The required direction is indicated by a direction tag in the
+#   mappings. A tag of <LR> means the corresponding Unicode character
+#   must have a strong left-right context, and a tag of <RL> indicates
+#   a right-left context.
+#
+#   For example, the mapping of 0x2B is given as <LR>+0x002B; the
+#   mapping of 0xAB is given as <RL>+0x002B. If we map an isolated
+#   instance of 0x2B to Unicode, it should be mapped as follows (LRO
+#   indicates LEFT-RIGHT OVERRIDE, PDF indicates POP DIRECTION
+#   FORMATTING):
+#
+#     0x2B ->  0x202D (LRO) + 0x002B (PLUS SIGN) + 0x202C (PDF)
+#
+#   When mapping several characters in a row that require direction
+#   forcing, the overrides need only be used at the beginning and end.
+#   For example:
+#
+#     0x24 0x20 0x28 0x29 -> 0x202D 0x0024 0x0020 0x0028 0x0029 0x202C
+#
+#   If neutral characters that require direction forcing are already
+#   between strong-direction characters with matching directionality,
+#   then direction overrides need not be used. Direction overrides are
+#   always needed to map the right-left digits at 0xB0-0xB9.
+#
+#   When mapping from Unicode to Mac OS Arabic, the Unicode
+#   bidirectional algorithm should be used to determine resolved
+#   direction of the Unicode characters. The mapping from Unicode to
+#   Mac OS Arabic can then be disambiguated by the use of the resolved
+#   direction:
+#
+#     Unicode 0x002B -> Mac OS Arabic 0x2B (if L) or 0xAB (if R)
+#
+#   However, this also means the direction override characters should
+#   be discarded when mapping from Unicode to Mac OS Arabic (after
+#   they have been used to determine resolved direction), since the
+#   direction override information is carried by the code point itself.
+#
+#   Even when direction overrides are not needed for roundtrip
+#   fidelity, they are sometimes used when mapping Mac OS Arabic
+#   characters to Unicode in order to achieve similar text layout with
+#   the resulting Unicode text. For example, the single Mac OS Arabic
+#   ellipsis character has direction class right-left,and there is no
+#   left-right version. However, the Unicode HORIZONTAL ELLIPSIS
+#   character has direction class neutral (which means it may end up
+#   with a resolved direction of left-right if surrounded by left-right
+#   characters). When mapping the Mac OS Arabic ellipsis to Unicode, it
+#   is surrounded with a direction override to help preserve proper
+#   text layout. The resolved direction is not needed or used when
+#   mapping the Unicode HORIZONTAL ELLIPSIS back to Mac OS Arabic.
+#
+#   2. Mapping the Mac OS Arabic digits
+#
+#   The main table below contains mappings that should be used when
+#   strict round-trip fidelity is required. However, for numeric
+#   values, the mappings in that table will produce Unicode characters
+#   that may appear different than the Mac OS Arabic text displayed on
+#   a Mac OS system using WorldScript. This is because WorldScript
+#   uses context-dependent display for the 0x30-0x39 digits.
+#
+#   If roundtrip fidelity is not required, then the following
+#   alternate mappings should be used when a sequence of 0x30-0x39
+#   digits - possibly including 0x2C and 0x2E - occurs in an Arabic
+#   context (that is, when the first "strong" character on either side
+#   of the digit sequence is Arabic, or there is no strong character):
+#
+#     0x2C	0x066C	# ARABIC THOUSANDS SEPARATOR
+#     0x2E	0x066B	# ARABIC DECIMAL SEPARATOR
+#     0x30	0x0660	# ARABIC-INDIC DIGIT ZERO
+#     0x31	0x0661	# ARABIC-INDIC DIGIT ONE
+#     0x32	0x0662	# ARABIC-INDIC DIGIT TWO
+#     0x33	0x0663	# ARABIC-INDIC DIGIT THREE
+#     0x34	0x0664	# ARABIC-INDIC DIGIT FOUR
+#     0x35	0x0665	# ARABIC-INDIC DIGIT FIVE
+#     0x36	0x0666	# ARABIC-INDIC DIGIT SIX
+#     0x37	0x0667	# ARABIC-INDIC DIGIT SEVEN
+#     0x38	0x0668	# ARABIC-INDIC DIGIT EIGHT
+#     0x39	0x0669	# ARABIC-INDIC DIGIT NINE
+#
+# Details of mapping changes in each version:
+# -------------------------------------------
+#
+#   Changes from version n03 to version n07:
+#
+#   - Change mapping for 0xC0 from U+066D to U+274A.
+#
+#   - Add direction overrides (required directionality) to mappings
+#     for 0x25, 0x2C, 0x3B, 0x3F.
+#
+##################
+0x0000 - 0x007F = 0x00 -
+0x00A0 = 0x81
+0x00AB = 0x8C
+0x00BB = 0x98
+0x00C4 = 0x80
+0x00C7 = 0x82
+0x00C9 = 0x83
+0x00D1 = 0x84
+0x00D6 = 0x85
+0x00DC = 0x86
+0x00E0 = 0x88
+0x00E1 = 0x87
+0x00E2 = 0x89
+0x00E4 = 0x8A
+0x00E7 = 0x8D
+0x00E8 = 0x8F
+0x00E9 = 0x8E
+0x00EA = 0x90
+0x00EB = 0x91
+0x00ED = 0x92
+0x00EE = 0x94
+0x00EF = 0x95
+0x00F1 = 0x96
+0x00F3 = 0x97
+0x00F4 = 0x99
+0x00F6 = 0x9A
+0x00F7 = 0x9B
+0x00F9 = 0x9D
+0x00FA = 0x9C
+0x00FB = 0x9E
+0x00FC = 0x9F
+0x060C = 0xAC
+0x061B = 0xBB
+0x061F = 0xBF
+0x0621 = 0xC1
+0x0622 = 0xC2
+0x0623 = 0xC3
+0x0624 = 0xC4
+0x0625 = 0xC5
+0x0626 = 0xC6
+0x0627 = 0xC7
+0x0628 = 0xC8
+0x0629 = 0xC9
+0x062A = 0xCA
+0x062B = 0xCB
+0x062C = 0xCC
+0x062D = 0xCD
+0x062E = 0xCE
+0x062F = 0xCF
+0x0630 = 0xD0
+0x0631 = 0xD1
+0x0632 = 0xD2
+0x0633 = 0xD3
+0x0634 = 0xD4
+0x0635 = 0xD5
+0x0636 = 0xD6
+0x0637 = 0xD7
+0x0638 = 0xD8
+0x0639 = 0xD9
+0x063A = 0xDA
+0x0640 = 0xE0
+0x0641 = 0xE1
+0x0642 = 0xE2
+0x0643 = 0xE3
+0x0644 = 0xE4
+0x0645 = 0xE5
+0x0646 = 0xE6
+0x0647 = 0xE7
+0x0648 = 0xE8
+0x0649 = 0xE9
+0x064A = 0xEA
+0x064B = 0xEB
+0x064C = 0xEC
+0x064D = 0xED
+0x064E = 0xEE
+0x064F = 0xEF
+0x0650 = 0xF0
+0x0651 = 0xF1
+0x0652 = 0xF2
+0x0660 = 0xB0
+0x0661 = 0xB1
+0x0662 = 0xB2
+0x0663 = 0xB3
+0x0664 = 0xB4
+0x0665 = 0xB5
+0x0666 = 0xB6
+0x0667 = 0xB7
+0x0668 = 0xB8
+0x0669 = 0xB9
+0x066A = 0xA5
+0x066D = 0xC0
+0x0679 = 0xF4
+0x067E = 0xF3
+0x0686 = 0xF5
+0x0688 = 0xF9
+0x0691 = 0xFA
+0x0698 = 0xFE
+0x06A4 = 0xF7
+0x06AF = 0xF8
+0x06BA = 0x8B
+0x06D2 = 0xFF
+0x06D5 = 0xF6
+0x2026 = 0x93
+0x274A = 0xC0
+END_MAP