Python subprocess.popen和rdiff-备份

Question

我想在python中创建一个rdiff备份包装程序，用于将windows机器备份到linux服务器。

我想在包装程序中处理rdiff备份的输出。但是，当使用子流程模块执行rdiff备份和将stdout和stderr传输到包装器时，stdout总是以管道的结尾结束。

在没有管道的情况下使用子进程时，stdout和stderr将以正确的顺序在控制台中打印。

我还注意到，在本地使用rdiff备份而不使用ssh管道时，stdout和stderr的顺序是正确的。Rdiff-backup还使用subprocess.popen打开ssh会话并将数据传输到服务器。我怀疑由于某种原因，stdout会被阻塞，直到ssh会话关闭。

下面是我的代码，这是一个简化的版本，真正的程序使用线程来侦听stdout：

import sys
import subprocess

class Rdiffbackup(object):
    def __init__(self):
        #self.io_q = Queue()
        self.exe = 'F:\\workspace\\pysubprocess\\bin\\rdiff-backup\\rdiff-backup.exe'

        self.verbosity = '-v5'

        self.ssh_exe = './bin/openssh/bin/ssh'
        self.quiet = '-q'
        self.compression = '-C'
        self.port = '-p 5555'
        self.key = '-i ./keys/rdiffbackup'
        self.options = '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
        self.remote_schema = self.ssh_exe +  ' ' + self.quiet + ' ' + self.compression + ' ' + self.port + ' ' + self.key + ' ' + self.options + ' %s rdiff-backup --server'  

    def start(self,source,dest):
        with subprocess.Popen([self.exe,self.verbosity,'--remote-schema',self.remote_schema,source,dest],stdout=subprocess.PIPE,stderr=subprocess.STDOUT) as self.proc:     
            for line in self.proc.stdout:
                sys.stdout.write(line.decode("utf-8"))      

if __name__ == '__main__':
    rdb = Rdiffbackup()
    source = "C:/Users/vdrmrt/Desktop/data"    
    dest = "vdrmrt@hostname::backup"
    rdb.start(source,dest)

产出：

-----------------------------------------------------------------
Detected abilities for source (read only) file system:
  Access control lists                         Off
  Extended attributes                          Off
  Windows access control lists                 On
  Case sensitivity                             Off
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
  Ownership changing                           Off
  Hard linking                                 On
  fsync() directories                          On
  Directory inc permissions                    On
  High-bit permissions                         On
  Symlink permissions                          Off
  Extended filenames                           On
  Windows reserved filenames                   Off
  Access control lists                         On
  Extended attributes                          On
  Windows access control lists                 Off
  Case sensitivity                             On
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Using rdiff-backup version 1.2.8
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server
Hardlinks disabled by default on Windows
Unable to import module xattr.
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data
Symbolic links excluded by default on Windows
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup

正确的输出：

Using rdiff-backup version 1.2.8
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server
Hardlinks disabled by default on Windows
Unable to import module xattr.
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
  Access control lists                         Off
  Extended attributes                          Off
  Windows access control lists                 On
  Case sensitivity                             Off
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
  Ownership changing                           Off
  Hard linking                                 On
  fsync() directories                          On
  Directory inc permissions                    On
  High-bit permissions                         On
  Symlink permissions                          Off
  Extended filenames                           On
  Windows reserved filenames                   Off
  Access control lists                         On
  Extended attributes                          On
  Windows access control lists                 Off
  Case sensitivity                             On
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Symbolic links excluded by default on Windows
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup

Answer 1

我终于解决了这个问题。

Rdiff-备份缓冲stdout和stderr时，它是管道到另一个程序。解决方案是使用额外的py2exe选项“unbuffered”：True重新构建rdiff备份。

我还必须添加一个额外的选项，以使其工作时，建设与windows 7。

Py2exe - win32api.pyc ImportError DLL load failed

下面是来自rdiff备份构建的最终setup.py的片段。

if '--single-file' in sys.argv[1:]:
            sys.argv.remove('--single-file')
            extra_options.update({
                'options': {'py2exe': {'bundle_files': 1,
                                       'unbuffered': True,
                                       'dll_excludes': [ "mswsock.dll", "powrprof.dll" ]}},
                'zipfile': None
            })

Answer 2

尝试为stdout和stderr使用单独的管道。先读stdout行，然后读stderr中的行。