Perl6多线程突发问题

Question

我编写了生成“随机”文本文件的程序，其中3个单词被存储在$keysfilename文件中的键替换。键文件具有非常简单的结构，如

ASD123ASD
QWE123QWE
XZC123ZXC

例如，当我使用多个线程时会出现问题。

my @threads = (^32).map({

如果在任意文件上失败，则出现错误。

    started
Thread<17>(14) got 1
Thread<18>(15) got 2
Thread<20>(17) got 17
Thread<5>(2) got 3
Thread<16>(13) got 4
Thread<21>(18) got 5
Thread<3>(0) got 6
Thread<8>(5) got 7
Thread<12>(9) got 10
Thread<11>(8) got 8
Thread<9>(6) got 9
Thread<14>(11) got 11
Thread<15>(12) got 12
Unhandled exception: Failed to open file C:\c\perltests\00000017.txt: no such file or directory
Thread<10>(7) got 13
Thread<13>(10) got 14
Thread<7>(4) got 15
Thread<19>(16) got 16
Thread<4>(1) got 0
Thread<6>(3) got 18
Thread<22>(19) got 19
   at <unknown>:1  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:4294967295)
Thread<23>(20) got 20
Thread<24>(21) got 21
 from gen/moar/m-CORE.setting:22337  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:34)
Thread<26>(23) got 22
Thread<25>(22) got 23
Thread<27>(24) got 24
 from gen/moar/m-CORE.setting:22374  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:FALLBACK:35)
Thread<28>(25) got 25
Thread<29>(26) got 26
 from gen/moar/m-Metamodel.nqp:3041  (C:\rakudo\share\nqp\lib/Perl6/Metamodel.moarvm::13)
Thread<30>(27) got 27
Thread<16>(13) got 28
Thread<17>(14) got 29
Thread<5>(2) got 30
Thread<18>(15) got 31
Thread<14>(11) got 32
Thread<15>(12) got 33
 from gen/moar/m-CORE.setting:25189  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:226)
Thread<30>(27) got 58
Thread<29>(26) got 57
Unhandled exception: Failed to open file C:\c\perltests\00000028.txt: no such file or directory
Thread<28>(25) got 56
 from gen/moar/m-CORE.setting:25203  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:92)
Thread<25>(22) got 55
 from gen/moar/m-CORE.setting:25199  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:44)
Thread<27>(24) got 54
 from gen/moar/m-CORE.setting:25506  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:96)
Thread<26>(23) got 53
 from gentexts.pl:54  (<ephemeral file>::189)
Thread<24>(21) got 52
Thread<23>(20) got 51
Unhandled exception: Failed to open file C:\c\perltests\00000058.txt: no such file or directory
Thread<6>(3) got 50
Thread<22>(19) got 49
   at <unknown>:1  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:4294967295)
Thread<34>(31) got 48
 from gen/moar/m-CORE.setting:22337  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:34)
Thread<33>(30) got 47
Thread<4>(1) got 46
 from gen/moar/m-CORE.setting:22374  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:FALLBACK:35)
Thread<7>(4) got 45
 from gen/moar/m-Metamodel.nqp:3041  (C:\rakudo\share\nqp\lib/Perl6/Metamodel.moarvm::13)
Thread<19>(16) got 44
Thread<11>(8) got 43
   at <unknown>:1  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:4294967295)
Thread<8>(5) got 42
 from gen/moar/m-CORE.setting:22337  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:34)
Thread<12>(9) got 41
Thread<10>(7) got 40
 from gen/moar/m-CORE.setting:25189  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:226)
Thread<13>(10) got 39
 from gen/moar/m-CORE.setting:25203  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:92)
Thread<31>(28) got 37
 from gen/moar/m-CORE.setting:25199  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:44)
Thread<32>(29) got 38
Thread<9>(6) got 36
 from gentexts.pl:44  (<ephemeral file>::15)
Thread<3>(0) got 35
Thread<21>(18) got 34
 from gen/moar/m-CORE.setting:22374  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:FALLBACK:35)
Thread<17>(14) got 59
 from gen/moar/m-CORE.setting:25506  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:96)
Thread<5>(2) got 60
Thread<18>(15) got 61
 from gentexts.pl:54  (<ephemeral file>::189)
Thread<9>(6) got 85
Thread<32>(29) got 84
 from gen/moar/m-CORE.setting:30638  (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:THREAD-ENTRY:44)

系统Win 10 x32，Rakudo 6.c

my $keysfilename := 'C:/c/keysfile.txt';
my $output       := 'C:/c/perltests';

my @keys = ();
for $keysfilename.IO.words {
    @keys.push($_);
}
my $len  := elems @keys;

my $lorem = q:to/END/; 
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec malesuada purus vel tincidunt eleifend. Fusce sollicitudin augue augue, et gravida dolor varius a. Vestibulum iaculis, dui iaculis iaculis molestie, tellus ante hendrerit massa, at volutpat risus metus vitae nisi. Integer neque magna, ultrices eu erat at, efficitur sollicitudin sem. Aliquam sed purus malesuada, porta est eu, rutrum neque. Quisque dolor leo, condimentum non mollis eget, tristique eget odio. Donec dignissim magna nec imperdiet iaculis. Vestibulum lorem ligula, euismod ac porttitor faucibus, rutrum eu ex.

Donec scelerisque nisi eget risus condimentum ultrices. Integer porta maximus quam, in lobortis quam fermentum eu. Morbi eu ligula consequat, aliquam sem eget, sollicitudin eros. Suspendisse potenti. Cras finibus metus et eros accumsan, id vehicula libero lobortis. Aenean vulputate lacinia urna at fringilla. Nulla id tincidunt lectus, quis accumsan lorem. In posuere magna non purus hendrerit rutrum. Maecenas in mattis tellus. Maecenas vel feugiat enim. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin convallis dapibus tellus vitae euismod. Nam eleifend dui quam, eget lobortis quam pulvinar id. Cras euismod posuere dolor non ultricies.

Ut dapibus porta faucibus. Duis velit ante, tincidunt id velit id, imperdiet egestas velit. Morbi efficitur enim dignissim interdum egestas. Vivamus eu urna condimentum, aliquam orci non, ullamcorper est. Phasellus egestas at tellus nec tristique. Fusce feugiat commodo faucibus. In hac habitasse platea dictumst. Quisque dignissim, mauris a pellentesque dictum, mauris velit tincidunt lorem, sed tincidunt libero enim vitae orci. Nam interdum, ante nec consequat vulputate, nisi turpis euismod nibh, sit amet elementum nunc diam non eros. Proin quis viverra risus. Vestibulum vestibulum diam in velit consectetur, eu elementum lacus sagittis. Morbi accumsan ac ante eget faucibus. In nec elit bibendum, tristique enim non, sodales ex. Donec sodales erat vitae odio cursus commodo.

Vestibulum felis lacus, mattis eget porta eget, mattis ut felis. Pellentesque aliquet, purus eu semper suscipit, sem ipsum euismod nunc, sed dapibus augue sem vel elit. Etiam tincidunt arcu ut nisi tempor commodo. Mauris at eros tincidunt, fringilla erat nec, sagittis ante. Integer et malesuada quam. Cras vel porta erat, sit amet efficitur erat. Praesent blandit purus quis urna consectetur, eget ultricies ipsum pulvinar. Phasellus ac molestie elit. Vestibulum et tincidunt velit. Aliquam a venenatis ipsum, nec commodo libero. Nullam eget consectetur lectus. Morbi placerat interdum erat nec interdum.

Morbi bibendum dui eu turpis pretium, eget aliquet augue aliquam. Aliquam eu dignissim mauris, vitae placerat augue. Ut sed tortor sit amet augue imperdiet rutrum. Aliquam erat volutpat. Morbi a turpis in sapien ultrices tristique. Proin quis vestibulum lorem, ut pharetra ex. Quisque tempor bibendum purus ac vehicula. Suspendisse tellus ipsum, imperdiet id sodales vel, congue a leo. Nulla gravida tincidunt nisi eu tempor. Mauris imperdiet tempor ante eget rutrum. Nam ut dui at augue laoreet mollis. Sed metus elit, viverra ac fringilla vel, fermentum et magna. Nam ligula purus, pretium vel dignissim vitae, fermentum at urna. Nullam ac ullamcorper felis. Maecenas dapibus consequat mi. 
END

my @words = $lorem.split(' ');
my $wordlen = @words.elems;

my &getNext = sub {
    my $counter = 0;
    my Lock $lock .= new;
    return sub (@filename) {
        $lock.lock;
        if ($counter < 100_000) {
            @filename[0] = $counter;
            $counter++;
        }
        $lock.unlock;
    };
}();

say "started";

my @threads = (^1).map({
    Thread.start(
        name => $_,
        sub {
            loop {              
                my @counter = (-1);
                getNext(@counter);
                last if @counter[0] < 0;
                say $*THREAD ~ " got " ~ @counter[0];
                my @copy = @words.clone;
                for (^3) {
                    my $pos = $wordlen.rand.round;
                    @copy[$pos] = @keys[$len.rand.round];
                }
                spurt sprintf($output ~ '/%08d.txt', @counter[0]), @copy.join(' ');
            }
        }
    );
}).join;

Answer 1

几个音符。

这是：

my @keys = ();
for $keysfilename.IO.words {
    @keys.push($_);
}

真的是这样吗：my @keys = $keysfilename.IO.words;

这个：@keys[$len.rand.round]真的就是这个：@keys.pick

我不太清楚你在用计数器数组做什么，好像你只是想要文件0..99999。

您正在为每个线程在子例程中设置锁。如果您希望防止多个线程访问某项内容，请在外部设置一个锁，并让每个线程使用它锁定独占代码。

您还在使用Thread来创建线程，但这实际上是一个非常低级别的接口。如果可能的话，尝试在降到该级别之前使用更高级别的结构。

在这种情况下，您似乎只想将100000个文件创建分成多个线程，以获得一些并行性。有几个构造使得这非常容易，hyper和race。由于您甚至不关心文件是以什么顺序创建的，所以您只需使用race，它将尽可能快地生成它们。

您可以使用race的“程度”参数来表示要使用多少线程。(您可能需要设置RAKUDO_MAX_THREADS以获得您想要的行为)。

my $keysfilename := 'C:/c/keysfile.txt';
my $output       := 'C:/c/perltests';

my @keys = $keysfilename.IO.words;

my $lorem = q:to/END/; 
Lorem ipsum ...
END

my @words = $lorem.split(' ');
my $wordlen = @words.elems;

[^100000].race(degree => 32).map({ 
#    say $*THREAD ~ " got " ~ $_;

    my @copy = @words;

    for (^3) {
        my $pos = $wordlen.rand.round;
        @copy[$pos] = @keys.pick;
    }

    spurt sprintf($output ~ '/%08d.txt', $_), @copy.join(' ');
});

Answer 2

在Perl 6中直接使用线程几乎是没有理由的，有许多为您做艰苦工作的特性。在大多数情况下，最简单的改进是只使用start而不是Thread.start，使用await LIST代替LIST».join或LIST.map(*.join)。

您调用的是List.join而不是Thread.join，这可能至少是您出现问题的部分原因。

有更多的事情使这很难阅读和理解。

从您编写getNext的方式来看，您似乎只知道或者大部分只知道JavaScript。我就是这样写的。

sub get-next (@filename) {
    state Int $counter = 0;  # set to zero only the first time it is called
    state Lock $lock .= new;

    $lock.protect: ->{
        if $counter < 100_000 {
            @filename[0] = $counter++;
        }
    }
}

但是，您只使用@filename来绕过参数的默认只读语义。

sub get-next ($filename is rw) {
    state Int $counter = 0;
    state Lock $lock .= new;

    $lock.protect: ->{
        if $counter < 100_000 {
            $filename = $counter++;
        }
    }
}

不过，只返回值就更有意义了。

sub get-next () {
    state Int $counter = 0;
    state Lock $lock .= new;

    $lock.protect: ->{
        $counter++ if $counter < 100_000
    } // Nil
    # the `if` will return `Empty` when $counter gets too big
    # but we want `Nil` instead
}

这要么返回计数器，要么返回Nil，这是一个未定义的值，因此您可以这样使用它：

loop {
    my $counter = get-next() orelse last;
    …
}

因为您只在循环中使用这个子程序，所以它可以调用last本身。

或者更好的方法是将声明计数器的行改为

state Int $counter = 0 but True;

然后可以将loop更改为while循环，而无需在任何地方添加last。

while get-next() -> $counter { … }

现在我已经向您展示了如何改进getNext，我将抛弃它，使用Perl 6的更好的特性。

(顺便说一句，使用频道更有意义)

# will probably still work with `use v6.c;`
# but v6.d has a better system backing `await`
use v6.d.PREVIEW;

# override the original default number of threads
# (16 threads currently)
BEGIN %*ENV<RAKUDO_MAX_THREADS> //= 32;
# the "correct" way to do this is setting $*SCHEDULER
# but this is easier

my $keys-filename = 'C:/c/keysfile.txt';
my $output-dir    = 'C:/c/perltests';

my @keys = $keys-filename.IO.words;

my @lorem = q:to/EOF/.split(' ');
…
EOF

say 'started';
END say 'finished';

for race ^100_000 -> $counter {
    say $*THREAD, " got ", $counter;
    my @copy = @lorem; # no need for .clone

    for (^+@copy).pick(3) -> $pos {
        @copy[$pos] = @keys.roll;
    }

    spurt $*SPEC.catfile($output-dir, $counter.fmt('/%08d.txt')), @copy.join(' ');
}

(我做了一次测试，它从来没有给出高于6的线程id )

我使用race而不是hyper，因为返回值无论如何都会被丢弃。

如果它不起作用的话，你就有了一个Rakudo的buggy版本。如果是这样，我建议更新到最新版本v2017.02

(say $*PERL.compiler.version;)

+@array与@array.elems相同

^ NUMBER与0 ..^ NUMBER相同，后者是糖的

Range.new( 0, NUMBER, :excludes-max )

这仍然存在一些“单词”有额外字符的问题。

例如，amet, elit.和ex.␤␤Donec

…

my @pos = ($lorem ~~ m:ex/ « \w+: /).map: { .from, .chars }
# @pos looks something like [(0,5),(6,5),(12,5),(18,3),…]

…

for race ^100_000 -> $counter {
    say $*THREAD, " got ", $counter;

    # vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
    my $copy = $lorem;

    # sort so that the transforms are done from the end of the string
    # towards the beginning of the string
    for @pos.pick(3).sort.reverse {
      $copy.substr-rw( |$_ ) = @keys.roll;
    }
    # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    spurt $*SPEC.catfile($output-dir, $counter.fmt('/%08d.txt')), $copy;
}