如何将深度学习算法与tcp传输视频服务集成

Question

如何通过tcp协议将人工智能处理的实时视频传输到其他平台.我将使用计算机作为服务器来接受视频，而Raspberry Pi作为客户端来传输视频。

我在tcp协议中使用OpenCV来实现实时视频传输，并且有一个完整可行的对象识别代码(来自https://github.com/PINTO0309/MobileNet-SSD-RealSense，感谢工程师PINTO0309)，.I在计算机端接收视频(Windows系统)，对象识别和视频传输在Raspberry Pi中。

服务器

import socket,time,cv2,numpy

def ReceiveVideo():
    address = ('193.169.4.155', 50000)
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind(address)

    s.listen(1)

    def recvall(sock, count):
        buf = b''
        while count:
            newbuf = sock.recv(count)
            if not newbuf: return None
            buf += newbuf
            count -= len(newbuf)
        return buf

    conn, addr = s.accept()
    print('connect from:'+str(addr))
    while 1:
        start = time.time()
        length = recvall(conn,16)
        stringData = recvall(conn, int(length))
        data = numpy.frombuffer(stringData, numpy.uint8)
        decimg=cv2.imdecode(data,cv2.IMREAD_COLOR)
        cv2.imshow('SERVER',decimg)

        end = time.time()
        seconds = end - start
        fps  = 1/seconds;
        conn.send(bytes(str(int(fps)),encoding='utf-8'))
        k = cv2.waitKey(10)&0xff
        if k == 27:
            break
    s.close()
    cv2.destroyAllWindows()

if __name__ == '__main__':
    ReceiveVideo()

客户端

import socket,cv2,numpy,time,sys

def SendVideo():
    address = ('193.169.4.155', 50000)
    sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    sock.connect(address)

    global capture

    capture = cv2.VideoCapture(0)
    ret, frame = capture.read()
    encode_param=[int(cv2.IMWRITE_JPEG_QUALITY),15]

    while ret:

        time.sleep(0.01)

        result, imgencode = cv2.imencode('.jpg', frame , encode_param)
        data = numpy.array(imgencode)
        stringData = data.tostring()

        sock.send(str.encode(str(len(stringData)).ljust(16)));

        sock.send(stringData);

        receive = sock.recv(1024)
        if len(receive):print(str(receive,encoding='utf-8'))

        ret, frame = capture.read()
        if cv2.waitKey(10) == 27:
            break
    sock.close()
if __name__ == '__main__':
    SendVideo()

对象标识码来自https://github.com/PINTO0309/MobileNet-SSD-RealSense/blob/master/MultiStickSSDwithPiCamera_OpenVINO_NCS2.py

在多次修改代码后，我遇到了以下问题。

1. 视频卡传输后处于第一帧。
2. 无法传输，Raspberry Pi显示错误为[Errno:32]Broken pipe，pc端显示错误。

OpenCV(3.4.2) C:\projects\opencv-python\opencv\modules\highgui\src\window.cpp:356: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'cv::imshow'

我认为第二个错误意味着没有得到图片。

最后，我想要实现识别视频的实时传输到计算机.

Answer 1

从视频中获取帧的逻辑是错误的:我在这里添加了一些注释。

# you take one frame here
ret, frame = capture.read()
encode_param=[int(cv2.IMWRITE_JPEG_QUALITY),15]

# this will loop without taking any new frame
while ret:

    time.sleep(0.01)
    # as result: at the first frame will be encoded successfully
    # but from second times, frame will be double/tripple encoded and became un-usable
    result, imgencode = cv2.imencode('.jpg', frame , encode_param)
    data = numpy.array(imgencode)
    stringData = data.tostring()

如何修复:将代码更改为如下内容

encode_param=[int(cv2.IMWRITE_JPEG_QUALITY),15]

while(True):
    ret,frame = cap.read()
    result, imgencode = cv2.imencode('.jpg', frame , encode_param)
    data = numpy.array(imgencode)
    stringData = data.tostring()