将UTF-8字节流转换为Unicode。

Question

如何轻松地创建从UTF-8字节流到Unicode代码点数组的映射？为了澄清，例如，如果我有字节序列：

c3 a5 76 aa e2 82 ac

映射应该产生两个长度相同的数组:一个具有UTF-8字节序列，另一个具有相应的Unicode编码点。然后，数组可以并排打印如下：

UTF8                UNICODE             
----------------------------------------
C3 A5               000000E5            
76                  00000076            
AA                  0000FFFD            
E2 82 AC            000020AC            

Answer 1

与流一起工作的解决方案：

use READ_SIZE => 64*1024;

my $buf = '';
while (1) {
   my $rv = sysread($fh, $buf, READ_SIZE, length($buf));
   die("Read error: $!\n") if !defined($rv);
   last if !$rv;

   while (length($buf)) {
      if ($buf =~ s/
         ^
         ( [\x00-\x7F]
         | [\xC2-\xDF] [\x80-\xBF]
         | \xE0        [\xA0-\xBF] [\x80-\xBF]
         | [\xE1-\xEF] [\x80-\xBF] [\x80-\xBF]
         | \xF0        [\x90-\xBF] [\x80-\xBF] [\x80-\xBF]
         | [\xF1-\xF7] [\x80-\xBF] [\x80-\xBF] [\x80-\xBF]
         )
      //x) {
         # Something valid
         my $utf8 = $1;
         utf8::decode( my $ucp = $utf8 );
         handle($utf8, $ucp);
      }

      elsif ($buf =~ s/
         ^
         (?: [\xC2-\xDF]
         |   \xE0            [\xA0-\xBF]?
         |   [\xE1-\xEF]     [\x80-\xBF]?
         |   \xF0        (?: [\x90-\xBF] [\x80-\xBF]? )?
         |   [\xF1-\xF7] (?: [\x80-\xBF] [\x80-\xBF]? )?
         )
         \z
      //x) {
         # Something possibly valid
         last;
      }

      else {
         # Something invalid
         handle(substr($buf, 0, 1, ''), "\x{FFFD}");
      }
}

while (length($buf)) {
   handle(substr($buf, 0, 1, ''), "\x{FFFD}");
}

上面的U+FFFD只返回Encode::decode('UTF-8', $bytes)认为不正确的内容。换句话说，它只在遇到以下情况时才返回U+FFFD：

• 一个意外的连续字节。
• 一个开始字节，后面没有足够的连续字节。
• “超长”编码的第一个字节。

仍然需要解码后检查才能返回U+FFFD，否则Encode::decode('UTF-8', $bytes)认为这是非法的。

Answer 2

编码有一个用于增量解码的API，但是它是没有文档的，您的里程可能会有所不同！它被编码：：编码和PerlIO：：编码的子类使用。与任何无文档化的API一样，它在任何时候都是可以更改的。一直以来，记录API都在努力。

#!/usr/bin/perl
use strict;
use warnings;

use Encode qw[STOP_AT_PARTIAL];

my $encoding = Encode::find_encoding('UTF-8');

my @octets = map { pack 'C', hex } qw<C3 A5 76 AA E2 82 AC F0 9F 90 A2>;
my $buffer = '';
while (@octets) {
    my $octets = $buffer . shift @octets;

    printf "--> processing: <%s>\n", 
      join ' ', map { sprintf '%.2X', ord } split //, $octets;

    my $string = $encoding->decode($octets, STOP_AT_PARTIAL);

    $buffer = $octets;

    if (length $buffer) {
        printf "buffered code units: <%s>\n", 
          join ' ', map { sprintf '%.2X', ord } split //, $buffer;
    }

    if (length $string) {
        printf "received code points: <%s>\n",
          join ' ', map { sprintf 'U+%.4X', ord } split //, $string;
    }
}

输出：

--> processing: <C3>
buffered code units: <C3>
--> processing: <C3 A5>
received code points: <U+00E5>
--> processing: <76>
received code points: <U+0076>
--> processing: <AA>
received code points: <U+FFFD>
--> processing: <E2>
buffered code units: <E2>
--> processing: <E2 82>
buffered code units: <E2 82>
--> processing: <E2 82 AC>
received code points: <U+20AC>
--> processing: <F0>
buffered code units: <F0>
--> processing: <F0 9F>
buffered code units: <F0 9F>
--> processing: <F0 9F 90>
buffered code units: <F0 9F 90>
--> processing: <F0 9F 90 A2>
received code points: <U+1F422>

Answer 3

下面是一种方法(脚本将字节序列作为第一个命令行参数)：

use feature qw(say);
use strict;
use warnings;

use Encode;

my @hex = split " ", shift;
my $bytes = join '', map { chr hex } @hex;
my @abytes;
my @achr;
while (1) {
    my $str = decode( 'UTF-8', $bytes, Encode::FB_QUIET );
    if ( length $str > 0 ) {
        for my $char ( split //, $str ) {
            my $bytes = encode( "UTF-8", $char, Encode::FB_CROAK | Encode::LEAVE_SRC);
            push @abytes, $bytes;
            push @achr, $char;
        }
    }
    last if length $bytes == 0;
    push @abytes, substr $bytes, 0, 1;
    push @achr, chr 0xfffd;
    $bytes = substr $bytes, 1;
}

my $fmt = '%-20s%-20s';
say sprintf $fmt, qw(UTF8 UNICODE);
say "-" x 40;
for my $char ( @achr ) {
    my $bytes = shift @abytes;
    my $str1 = join ' ', map { sprintf '%X', ord $_} split //, $bytes;
    my $str2 = sprintf '%08X', ord $char;
    say sprintf $fmt, $str1, $str2;
}