根据空格字符和指定字符长度，有效地将长文档拆分为多个较短的文档。

Question

我需要编写一个函数，根据预先指定的字符长度将长文档拆分为空格字符(\s)上的较短文档。

例如，为了举例说明，我有一个文本文档，其中包含175,000,000字符(包括所有标点符号和空格字符)。我想把这份文件分成大约100000个字符的较短的文件。

当然，发生分裂的地点并不是100,000/200，00/300,000.字符，因为空白字符可能不在这些位置。如果空白字符不在所需的拆分点(例如，如果100000个字符不是空白字符)，则函数将查找与左最接近的惠特空格字符并在那里拆分。下面是我对这个函数的尝试，但是这个函数看起来非常慢。

whitespace_regex = re.compile(r"\s")

def foo(text):
    # If a document is 100000 character long or shorter
    # no splitting is needed
    if len(text) <= 100000:
        yield text
    # Splitting if a document is longer than 100000 characters
    elif len(text) > 100000:
        # A while loop until there is nothing left to be split
        while len(text) > 100000:
            # Split a document into two segments: 
            #     left: 100000 character long
            #     text: the rest of the document
            left, text = text[:100000], text[100000:]

            # Look for the rightmost whitespace character in the 'left'
            # segment by first reversing the string so that the whitespace
            # returned by the regex search is the rightmost whitespace
            whitespace = whitespace_regex.search(left[::-1])

            # Get the start index of the returned whitespace. If -index
            # is 0, then that means pro
            index = whitespace.start()
            index = -index
            # if the whitespace is not exactly at the desired position,
            # yield the part to the left of the whitespace character, and
            # combine the part of the left segment to the right of the
            # whitespace character with the rest of the remaining text
            if index < 0:
                text = left[index:] + text
                left = left[:index]
            yield left
        if text:
            yield text        

我在一个有175,000,000个字符的文档上测试了速度，然后用了将近6分钟的时间完成了对文档的拆分：

a = "John did what others told him to do" * 5000000
print(f"Document's length is {len(a)}")
#Document's length is 175000000

start_time = time.time()
segs = [x for x in foo(a)]
print(time.time() - start_time)
#344.3530957698822

，我想知道是否有一种方法可以编写一个更有效的函数来完成这个任务。

Answer 1

这个问题与文件的大小有很大的关系。例如，10倍小的文档在0.3秒内运行在我的机器上，您的大小在144.8中运行。我认为问题是，每次你从左边切掉一块，剩下的文字就会移动。因此，一种解决方案可能是从后面开始切割您的文档。另一种解决方案可以是将数组分割成较小的块，并在它们上运行函数：

a = "John did what others told him to do" * 5000000
print(f"Document's length is {len(a)}")
#Document's length is 1750000000

start_time = time()
segs = []
block_size = 17500000
start_ind = 0
num_blocks = int(len(a)/block_size) + 1
for i in range(num_blocks):
  if len(a)-start_ind > block_size:
      block = a[start_ind : start_ind + block_size]
  else:
      block = a[start_ind :]

  block_segs = [x for x in foo(block)]

  start_ind += block_size - len(block_segs[-1])

  segs += block_segs[:-1]
  if len(a)-start_ind > block_size: segs += block_segs[-1]


print(time() - start_time)

这大约需要3.0秒。又快又脏我想..。