System and Method of Transforming Information Between UCS and EBCDIC Representations Employing EBCDIC-friendly Transformation Formats

The present invention relates to a system and method of transforming information between different representations. More particularly, the present invention relates to the transformation of information represented in ISO/IEC 10646-1 (UCS) or Unicode encodings to and/or from EBCDIC-friendly representations.

With the exponential growth of the internet and computer networks in general, it has become increasingly important that the computers connected by these networks, and their operating systems, be able to work with information represented in a variety of character sets. For example, in North America the majority of computer systems represent information in Latin characters, while in Russia information is primarily represented in Cyrillic characters and in Japan information is often represented in Katakana characters.

Within computer systems and networks, characters are represented by values assigned to characters, the specific assignments being defined within a coded character set. A common encoding scheme is ASCII which, for example, specifies that the value 97 (hexadecimal 0x61) represents the lower case "a" character, the value 65 (0x41) represents an upper case "A" character, etc. Another common encoding scheme is EBCDIC which, for example, specifies that the value 129 (hexadecimal 0x81) represents the lower case "a" character and the value 193 (hexadecimal 0xC1) represents the uppercase "A" character.

While different character sets exist in both of these encoding schemes, other encoding schemes are employed in a variety of other countries and/or languages. Accordingly, using information represented in a different coded character set, or changing the representation of information from one coded character set to another, is often required but can be difficult to accomplish. In particular, ASCII and EBCDIC encoded character sets are single octet (byte) systems and thus can only encode a maximum of 256 characters, including all control characters, etc. When it is desired to encode a larger set of characters, a multi-octet encoding scheme is required.

In an attempt to define a universal method of representing characters which allows all, or substantially all, desired characters to be represented, the International Standards Organization (ISO/IEC) has published the Universal Multiple Octet Coded Character Set (UCS) as standard ISO/IEC 10646-1, and it's amendments. UCS can define over two billion characters and UCS includes a two-byte (UCS-2) form and a four-byte (UCS-4) form and these forms and the UCS standard are described in ISO/IEC 10646-1: 1993--Universal Multiple-Octet Coded Character Set and in the corresponding Unicode Standard V2.0, from the Unicode Consortium which employs the UCS-2 form of UCS, and the contents of these publications are incorporated by reference herein.

While UCS provides a reasonable framework within which to handle large numbers of characters, many pre-UCS computer systems and pre-UCS information/data employs other character sets and it is necessary to transform information represented in UCS to a form compatible with these other character sets, or vice versa. In particular, information represented in UCS cannot be directly processed by systems using ASCII encoding or EBCDIC encoding, therefore transformation to and from UCS is required. Due to the widespread use of ASCII (7 bit, 128 characters), it was particularly desired to have a transform for UCS to ASCII and back.

A transform referred to as UTF-8 has been developed to transform UCS-2 to and from ASCII. This transform is defined in Amendment No. 2 to ISO/IEC 10646-1 and the contents of this publication are incorporated herein by reference. UTF-8 transforms UCS-2 information to an ASCII-friendly format and vice-versa and maintains the ASCII characters (assigned to values 0 through 127) as single octets and the rest of the UCS-2 character set as sequences of two to six octets (a multi-octet sequence). In particular, the 95 characters and the SPACE, commonly referred to as the graphic characters of the ASCII set, or the G0 set and the 33 control characters (the 32 control characters from 0x00 to 0x1F, referred to as the C0 set, and the DELETE character at 0x7F) are maintained as single octets.

International ISO/IEC 4873 standard defines an 8-bit encoding standard, which defines the structure for other ISO/IEC 8-bit characters sets, such as ISO/IEC 8859-1 (Latin Alphabet No. 1), referred to herein as ISO-8 bit encoding, which employs the same assignment of ASCII characters to values 0x00 through 0x7F but which also includes additional characters. In particular, ISO/IEC 4873 also includes a second set of control characters from 0x80 to 0x9F, referred to as the C1 set and these latter control characters are mirrored as two-octet escape sequences, i.e.--ESC xx, where `xx` is a suitable value. The UTF-8 transform does not maintain the C1 set as single-octet characters.

It should be emphasised that UTF-8 transforms information between UCS and ASCII-friendly representations, not ASCII representations. In particular, UTF-8 transforms UCS to an ASCII-friendly form wherein the characters assigned to the first 128 values of the ASCII character set are maintained as single octet strings, but all other characters are represented as variable-length multiple-octet strings of between two and six octets.

While UTF-8 is generally useful in many circumstances, it is not useful to transform information represented in UCS to an EBCDIC representation and/or vice versa. Unlike ASCII, EBCDIC includes 65 control codes, in addition to the 191 letter, number and punctuation characters (generally referred to as `graphic characters`). The graphics characters are located at 64 (0x40), for the SPACE character, and from 65 (0x41) to 254 (0xFE) and the control characters are assigned from 0x00 to 0x40 and the EO ("eight ones") character which is assigned to 0xFF.

Thus, for compatibility with computer systems using and/or information represented in EBCDIC, information transformed between UCS and EBCDIC-friendly forms must ensure that the 65 control characters be transformed to single-octet representations.

Further, despite the fact that there exist many EBCDIC coded character sets (commonly referred to as "code pages"), a subset of these characters, which is often referred to as the "invariant" character set, is assigned to common values in most or all code pages. For example, the upper case Latin characters "A" through "Z" appear at values 193 (0xC1) to 233 (0xE9) in all EBCDIC code pages. Similarly, the numerals "0" through "9" appear at values 240 (0xF0) through 249 (0xF9) in all EBCDIC code pages. While there are some exceptions to these invariant assignments, as discussed below in more detail, it is desired that a transform maintain the single-octet values to which the invariant character set is assigned.

It is an object of the present invention to provide a system and method for transforming information between UCS and EBCDIC-friendly representations. It is another object of the present invention to provide a system and method for transforming information between UCS and ISO-8 bit-friendly representations.

According to a first aspect of the present invention, there is provided a method of transforming information from a fixed-length multiple-octet representation to a target variable-length multiple-octet representation, the transformation ensuring that a selected set of characters are represented by single octets in said target representation, comprising the steps of:

(i) receiving an input character string in said fixed-length representation and transforming said input character string to a variable-length multiple-octet intermediate representation, said transformation producing a single octet representation in said intermediate representation for a pre-selected set of input characters in said fixed-length representation;

(ii) receiving a character string in said intermediate representation and transforming said character string to an equivalent number of octets in said target representation, said transformation maintaining characters represented by single octets in said intermediate representation as characters represented as single octets in said target second representation.

According to a second aspect of the present invention, there is provided a method of transforming information from a variable-length multiple-octet representation to a fixed-length multiple-octet representation, comprising the steps of:

(i) receiving an input character string in said variable-length representation and transforming said input character string to the same number of octets in a variable-length multiple-octet intermediate representation; and

(ii) examining said character string which has been transformed to said intermediate representation to determine the number of octets representing each character and transforming each character represented by said determined number of octets to a corresponding value in said fixed-length multiple octet representation.

According to another aspect of the present invention, there is provided a method of transforming information from a fixed-length multiple-octet representation to a variable-length multiple-octet representation by receiving an input character string in said fixed-length representation and transforming said input character string to said variable-length multiple-octet representation, said transformation producing a single octet representation in said variable-length representation for a pre-selected set of input characters in said fixed-length representation.

According to yet another aspect of the present invention, there is provided a system for transforming information between a fixed-length multiple-octet representation and a variable-length multiple-octet representation, comprising:

a first transform stage to transform information between said fixed-length multiple-octet representation and an intermediate variable-length multiple-octet representation, said first transform stage including a pre-defined set of characters which are represented as single octet values in said intermediate variable-length multiple octet representation; and

a second transform stage to transform information between said intermediate representation and said variable-length multiple-octet representation, said second transform stage including means to maintain characters represented as single octets in either of said intermediate representation and said variable-length multiple-octet representation as single octets in the other of said intermediate representation and said variable-length multiple-octet representation.

Preferably, the fixed-length multiple-octet representation is UCS-2, UCS-4 or the like. Also preferably, the variable-length multiple-octet intermediate representation is ISO-8 bit-friendly. Also preferably, the variable-length multiple-octet representation is EBCDIC-friendly.

Preferred embodiments of the present invention will now be described, by way of example only, with reference to the attached Figures, wherein:

FIG. 1 shows a block diagram of an embodiment of the present invention functioning as a UCS to EBCDIC transformation system and method; and

FIG. 2 shows a block diagram of the embodiment of FIG. 1 functioning as an EBCDIC to UCS transformation system and method.

An embodiment of the present invention which is useful for transforming information between UCS representations and EBCDIC-friendly representations is indicated generally at 20 in FIG. 1. As shown in the Figure, in a presently preferred embodiment of the invention the transformation is performed to two stages. A first stage 24 converts an input, referred to herein as a U-string (UCS-2 or UCS-4), to an intermediate form, referred to herein as an I8-string, and a second stage 28 converts the I8-string to an EBCDIC-friendly output, referred to herein as an En-string. FIG. 2 shows this embodiment 20 of the present invention functioning to transform information in an EBCDIC representation (En-string or EBCDIC) to a UCS representation.

As used herein, the term U-string is intended to comprise both UCS-4 (four octet values from 0 to 2,147,483,647 (0x7FFFFFFF)) and UCS-2 (two octet values from 0 to 65,535 (0xFFFF)) and UCS-2 with S-zones, (two pairs of octets). Each character in a U-string is represented as a fixed, multi-octet value, either four octets for UCS-4 or two octets for UCS-2 or two pairs of octets for UCS-2 with S-zones. As is known, the arrangement of UCS-4 is often discussed in terms of groups (indicated by the most significant octet of the four octets), planes (indicated by the next most significant octet) and row and column elements (indicated by the two least significant pair of octets). UCS-2 is discussed in terms of Row and Column elements and UCS-2 with S-zones is discussed in terms of High and Low pairs. Amendment 1 of ISO/IEC Standard 10646 and V2.0 of the Unicode standard discuss the UTF-16 transform whereby the High and Low pairs can be transformed to a UCS-4 representation of row and column elements in group 0, planes 1 through 16, as discussed below.

The Basic Multilingual Plane, or BMP, is 65,536 elements in group 0, plane 0 of UCS-4 (i.e. 0x00000000 to 0x0000FFFF) and most of the world's widely used characters have been allocated positions in the BMP. In a roadmap document adopted by ISO/IEC JTC1/SC2/WG2 and the Unicode Technical Committee, it is shown that in practice all anticipated characters can be represented in the combination of the BMP and the next two planes in group 0 (i.e. 0x00000000 to 0x0002FFFF).

UCS-2 is a subset of UCS-4 and its two octets represent the row and column elements of the BMP. UCS-2 with S-zones comprises, in addition to the BMP, planes 1 through 16 of group 0. As presently implemented, first stage 24 can process a U-string comprising UCS-4, UCS-2 or UCS-2 with S-Zones representations. In the event that invalid S-Zone pairs are detected, first stage 24 will process each pair of octets as a single value in the range 0x4000 to 0xFFFF.

The output of first stage 24 is an intermediate representation referred to herein as an I8-string. The I8-string is a variable length multi-octet sequence which can comprise 1 to 7 octets and is an ISO-8 bit-friendly format. As used herein, the term "friendly format" is intended to comprise any format whereby a multi-octet string has been arranged to represent a selected set of values as single octet values. Specifically, in I8-strings the first 127 characters (including the C0 set) of the ASCII character set and the C1 set of control characters, from the ISO-8 bit encoding scheme, are preserved as single octet values and other characters are represented as 2 to 7 octet values. Thus, processing of 18-strings by ASCII and/or ISO-8 bit systems is simplified and the format is ISO-8 bit-friendly.

For all I8-strings of two or more octets, the total number of octets in the string is indicated in the leading octet of the string by the number of high order bits followed by a zero. For example, b110xxxxx (where `x` represents a `1` or a `0` and the leading `b` indicates a binary representation) indicates that the I8-string has two octets and b11111110 indicates that the I8-string has seven octets. Each trailing octet after the leading octet begins with the bit sequence b101xxxxx. Thus, for example, a two octet I8-string might be b110xxxxx b101xxxxx. It should be noted that one exception exists to this in that the lead octet of a seven octet I8-string can be all 1's (i.e--0xFF) when the most significant bit of the corresponding U-string is a 1.

First stage 24 transforms a U-string input, in a specified format, to an I8-string output according to the table shown in Appendix A, for UCS-2, and according to the table shown in Appendix B for UCS-2 with the S-zone. The table shown in Appendix C is used for UCS-4. In the tables, the first two columns represent the lower and upper values of the U-string to which the table row applies, and these values are represented as 16 bit values in UCS-2 and 32 bit values in UCS-4, and the next column represents the number of octets in the corresponding I8-string. The next column in the table represents the number of bits from the U-string that are contained in the I8-string in the form M(n), where M represents the maximum number of bits that can be carried in the given number of octets assigned to the I8-string and (n) represents the number of bits which are varied to indicate the value within the From/To range for that table row. The next column shows the permitted values for the octet sequence for the I8-string, in binary form, with the lowest and greatest values shown, along with the bit pattern of the permitted values. The next column shows the lowest and greatest permitted octet sequence values in hexadecimal form and these correspond to the transformed values of the From and the To values in the first two columns.

Transformation from an I8-string back to a valid UCS string (UCS-2 or UCS-4) is accomplished by concatenating the values of the x bits shown in the appropriate table row, stripping off the leading 1's and 0's of each octet, and padding the resulting concatenated string with leading 0's to obtain a final UCS string of the desired two octet or four octet form.

As mentioned above, one exception to the conversion of I8-strings to U-strings exists in that, for I8-strings of seven octets wherein the lead byte has no zeros (0x40000000 to 0x7FFFFFFF), the least significant bit of the leading octet is maintained as the most significant bit of the U-string.

In UTF-8, for circumstances wherein it is possible to encode a value in two or more ways, the "shortest" possible encoded value is defined to be the correct value. This "shortest string rule" is also employed in the presently preferred embodiment of the invention.

The table in Appendix D shows the form of I8-string octets from the U-strings transformed by stage 24. In contrast, the table in Appendix E shows the form of I8-string octets from U-strings transformed by a prior art UTF-8 transform.

As will be apparent to those of skill in the art, first stage 24 transform data between fixed-length multi-octet (UCS-4 or UCS-2) formats and a multi-octet format of from one to seven octets. The transformation employed ensures that a selected set of characters are transformed to single octet values corresponding to ASCII/ISO-8 bit character set assignments, as defined in ISO/IEC 4873. It should be noted that the EBCDIC control set is not identical to the C0 and C1 sets of ISO/IEC 4873, but the number of control characters is the same (i.e. -65). A default correspondence, as defined in IBM's Character Data Representation Architecture (CDRA), as published by IBM, is maintained in a presently preferred embodiment of the present invention.

As will be apparent to those of skill in the art, while the I8-string is an intermediate representation in the transformation between UCS and EBCDIC-friendly representations, it is contemplated that I8-strings can also be directly employed in ASCII/ISO-8 bit-based systems. In such a circumstance, the present invention will transform information between UCS and the I8, ASCII/ISO-8 bit-friendly, format.

The output of second stage 28 of system 20 in FIG. 1 is referred to herein as an En-string, wherein the "E" refers to EBCDIC and the "n" refers to one of one or more predefined EBCDIC code pages. Like many other character representation systems, EBCDIC has different character assignments for use with various languages, and/or in various countries, etc. and these different character assignments are generally referred to as code pages. For example, EBCDIC code page 1047 (commonly referred to as the "EBCDIC Open Systems" code page) has different assignments and/or characters than does EBCDIC code page 37 (commonly referred to as the "US/Canada CECP" code page). In a preferred aspect of the present invention, a set of values can be defined for "n" which correspond to various EBCDIC code pages. For example, n=1 can be defined to identify code page 1047, n=2 can identify code page 500 (International #5), n=3 can identify code page 37, etc. Thus, stage 28 can perform a transformation between I8-strings and E1-strings, or E2-strings, etc.

Each En-string is a multi-octet transformed representation of the I8-string, and thus the U-string. In a presently preferred embodiment of the present invention, the actual conversion of I8-strings to an En-string is performed by table look up which has a unique one to one mapping between I8-string octets and En-string octets and which is symmetric, i.e.--when an I8-string is converted to an En-string and then converted back to an I8-string, the result is identical to the original I8-string. Thus, the transformation is performed in an octet by octet manner.

A character preservation principle is employed for the graphic characters of the I8-string (octet values in the range 0x00 to 0x9F), while an octet preservation principle is employed for the range 0xA0 to 0xFF. Specifically, the transformation employed in stage 28 preserves the single octet representation of all EBCDIC control characters by assigning I8-string octets in the ranges of 0x0 to 0x1F and 0x80 to 0x9F, and character 0x7F to EBCDIC octets in the range of 0x00 to 0x3F and to 0xFF. The transform also preserves the single octet representation for the set of 95 graphic characters (including the SPACE character) of the ASCII/ISO-8 bit character set at their allocated positions in the target EBCDIC code page (indicated by the selected value of `n`) and preserves the leading octets and trailing octets from the I8-string as their corresponding single octets in the En-string. Finally, the transform maintains the symmetry between the forward and reverse pairings. As the transform maintains the single octet representation of EBCDIC control characters and graphics characters, En-strings are EBCDIC-friendly.

The ease with which a determination can be made in stage 28 of whether a sequence is a single octet or multi-octet or whether an octet is a leading or trailing octet is dependent upon the target EBCDIC code page. The technique which is presently preferred is to use a reverse vector look-up to locate the I8-string and then determine the resultant octet using the I8-string rules by consulting a table, such as that shown in Appendix D. As an alternative, a shadow vector containing the category of each octet in the En-string can be constructed and employed.

As presently implemented, five target EBCDIC code page alternatives are available, although more can be implemented if desired. Specifically: n=1 corresponds to code page 1047, "EBCDIC Open Systems (Latin-1)"; n=2 corresponds to code page 500, "EBCDIC multi-lingual CECP (Latin-1)"; n=3 corresponds to code page 37, "EBCDIC US CECP (Latin-1)"; n=4 corresponds to a special alternative, described below; and n=5 corresponds to code page 290, "EBCDIC Katakana Extended". Complete definitions of the above-mentioned code pages are available from a variety of sources and will thus not be described further herein.

The differences for n=1, n=2 and n=3 are due to the mapping of a few variant characters, and these differences are listed in the table of Appendix F. In the table, GCGID refers to an ID defined for the glyph.

Table 1 in Appendix G shows a table for converting between single octet I8-strings and single octet E1-strings (code page 1047) and table 2 shows the categories of octets in E1-strings. Table 3 in Appendix G shows a table for converting between single octet I8-strings and single octet E2-strings (code page 500) and table 4 shows the categories of octets in E2-strings. Table 5 in Appendix G shows a table for converting between single octet I8-strings and single octet E3-strings (code page 37) and table 6 shows the categories of octets in E3-strings. As will be apparent to those of skill in the art, to transform an I8-string to an E1, E2 or E3-string, a reverse transform-vector lookup is employed and range checks (using the table in Appendix F, for example) are performed to determine if the I8-string is a single octet, leading octet or trailing octet, and how many octets are in the string if it is a multi-octet string. Single octet conversions are then performed using the appropriate ones of tables 1 through 6 of Appendix G.

In UCS, the values 0xFFFE and 0xFFFF are not used for character allocation in any plane. Specifically, 0xFFFE is reserved for use as a Signature and 0xFFFF is used to represent a numeric value that is guaranteed not to be a character. 0xFFFF is often used for purposes such as indicated the end of an index of characters, etc. While UTF-8 avoids the use of 0xFE and 0xFF in it's octets, the I8-strings produced by stage 24 and En-strings produced by stage 28 do use the 0xFE and 0xFF values. Specifically, in the I8-string, 0xFE and 0xFF are lead octets of seven-octet I8-strings but they are surrounded by a value less than 0xC0. Thus, neither 0xFFFF or 0xFFFE sequences are valid in properly formed I8-strings. Depending upon the target code page selected, the I8-string to En-string pairings are 0xFE to 0x8E or 0x4A and 0xFF to 0xDF or 0xE1.

In the En-string, the values 0xFE and 0xFF are generated by converting I8-string values according the appropriate conversion table (depending upon the target code page). The I8-string to En-string pairings are 0xDA or 0xBF to 0xFE and 0x9F to 0xFF. 0xBF is the last element of the set of trailing octets possible in a multi-octet I8-string and must be preceded by a lead octet and zero or more trailing octets (all within the range of 0xA0 to 0xFF). An 0x9F cannot precede an 0xBF in a properly formed I8-string, and thus the sequence 0xFFFE cannot appear in an En-string. 0x9F is associated with a control character, specifically the "Application Program Command (APC)" character in ISO-8, C1. According to ISO/IEC 6429, APC is followed by a parameter string using values from 0x8 to 0x0D and from 0x20 to 0x7E and is terminated by the control function "String Terminator (ST)" which is associated with value 0x9C in ISO-8, C1. Thus, the sequence 0xFFFF (which would be equivalent to two APC characters without intervening parameters or ST characters) cannot appear in an En-string.

In the presently preferred embodiment of the invention, n=4 (i.e. E4-strings) have been reserved for special use. Specifically, when a table lookup method is employed in the transform of stage 28, the different octets of the E1, E2 or E3-string are not well grouped for checking. In the arrangement provided with n=4, a single octet to single octet vector lookup is employed such that octets of E4-strings can be examined directly. Table 1 of Appendix H shows the groupings for E4-strings and table 2 shows the distribution of octets from single octet E4-strings and table 3 shows the categories of the octets in E4-strings. Table 4 shows a table for converting between single octet I8-strings and single octet E4-strings. As shown, E4-strings employ the E1 mappings as a base for mapping single octets in the range 0x00-0x9F from the I8-string and the remaining space is allocated to group leading and trailing octets in contiguous ranges, where possible. Of course, the distribution of single octets is such that it is not always possible to keep each category in a contiguous range.

As will be apparent to those of skill in the art, if a different base than E1is selected for mapping single octets, the distribution of octets of the variant category will also change as will the distribution of the remaining octets.

As mentioned above, some few EBCDIC code pages do not maintain the mapping of the invariant character set. A widely used code page in which some of the characters in the invariant character set are assigned to different values is code page 290, commonly referred to as Katakana Extended. In this code page, the lower case Latin characters ("a" to "z") are not assigned to the ranges used in most other code pages, i.e.--0x81 to 0x89, 0x91 to 0x99 and 0xA2 to 0A9. Accordingly, the transformation to and/or from E5-strings has been designed to accommodate the different assignments of the "invariant" character set employed in code page 290. As will be apparent to those of skill in the art, additional special transformations and E-string definitions can be created, if required, for other code pages in which characters of the invariant character set have a variant assignment.

Table 1 in Appendix I shows a table of the groupings of E5-string octets and table 2 shows a table for converting between single octet I8-strings and single octet E5-strings (code page 290). Table 3 shows the categories of octets in E5-strings.

The present invention provides a novel system and method of allowing information and/or systems using one or more EBCDIC code pages to represent information to transparently access UCS-coded information and/or systems. The first stage of the invention transforms information between fixed-length multiple-octet representations, such as UCS-2or UCS-4, and a variable-length multiple-octet representation wherein the representation of a selected set of characters as single octets is ensured. The second stage of the invention transforms information from a variable-length multiple-octet representation to another variable-length multiple-octet representation. The transformation is performed on an octet by octet basis, and thus the number of octets representing a character in the output of the transform is the same as the number of octets representing the character in the input of the transform. The transform thus preserves single-octet characters as single octets through the transformation. Further, the transform can assign a selected set of invariant or other characters to selected values. While the above discussion concentrates on transformation between UCS and EBCDIC-friendly representations, the present invention is not so limited and can be employed to transform information between UCS and other variable-length multiple-octet representations. Similarly, the present invention can be employed to transform information between other fixed-length multiple-octet representations and variable-length multiple-octet representations.

The above-described embodiments of the invention are intended to be examples of the present invention and alterations and modifications may be effected thereto, by those of skill in the art, without departing from the scope of the invention which is defined solely by the claims appended hereto.