在webRTC中，在捕获程序中调用onFrame之后，没有从C++代码传输到浏览器的帧

Question

我正在尝试使我的应用程序与webrtc保持同步。这基本上是一个桌面流应用程序，使用webRTC将C++应用程序的视频流传输到浏览器。我的实现曾经使用过一大堆过时的东西，比如SignalFrameCaptured和cricket::CapturedFrame。现在看看webRTC，这些类/信号似乎不可用。

这是我的捕获者：

class Capturer
: public cricket::VideoCapturer,
  public webrtc::DesktopCapturer::Callback
{

 public:

   sigslot::signal1<Capturer*> SignalDestroyed;

   Capturer();
   ~Capturer();

   void ResetSupportedFormats(const std::vector<cricket::VideoFormat>& formats);
   bool CaptureFrame();
   cricket::CaptureState Start(const cricket::VideoFormat& format) override;
   void Stop() override;
   bool IsRunning() override;
   bool IsScreencast() const override;
   bool GetPreferredFourccs(std::vector<uint32_t>* fourccs) override;


   virtual void OnCaptureResult(webrtc::DesktopCapturer::Result result,

                         std::unique_ptr<webrtc::DesktopFrame> desktop_frame) override;


 private:
   bool running_;
   int64_t initial_timestamp_;
   int64_t next_timestamp_;
   std::unique_ptr<webrtc::DesktopCapturer> _capturer;

};

从另一个线程定期调用Capturer::CaptureFrame()，并使用DesktopFrame作为参数按预期调用Capturer::OnCaptureResult。现在来看一下OnCaptureResult的实现

void Capturer::OnCaptureResult(webrtc::DesktopCapturer::Result result,

                          std::unique_ptr<webrtc::DesktopFrame> desktopFrame)
{

  if (result != webrtc::DesktopCapturer::Result::SUCCESS)
  {
   return; // Never called. Which leads me to conclude thedesktop capturer works
  }


  int width = desktopFrame->size().width();
  int height = desktopFrame->size().height();
  rtc::scoped_refptr<webrtc::I420Buffer> res_i420_frame = webrtc::I420Buffer::Create(width, height);


  webrtc::ConvertToI420(webrtc::VideoType::kABGR,
                        desktopFrame->data(),
                        0, 0,
                        width, height,
                        0,
                        webrtc::kVideoRotation_0,
                        res_i420_frame);

  webrtc::VideoFrame frame = webrtc::VideoFrame(res_i420_frame, webrtc::kVideoRotation_0, next_timestamp_ / rtc::kNumNanosecsPerMicrosec);


  this->OnFrame(frame, width, height);
}

从chrome::webrtc-internals可以看出，任何帧都不会传输到连接的浏览器。

​

​

回到代码，下面是我如何创建peerConnection的：

void Conductor::connectToPeer() {

  this->_peerConnectionFactory = webrtc::CreatePeerConnectionFactory();

  if (!this->_peerConnectionFactory.get())
  {
     std::cerr << "Failed to initialize PeerConnectionFactory" << std::endl;
     throw new std::runtime_error("Cannot initialize PeerConnectionFactory");
  }   

  webrtc::PeerConnectionInterface::RTCConfiguration config;
  webrtc::PeerConnectionInterface::IceServer server;
  server.uri = "stun:stun.l.google.com:19302";
  config.servers.push_back(server);

  webrtc::FakeConstraints       constraints;constraints.AddOptional(webrtc::MediaConstraintsInterface::kEnableDtlsSrtp, "true");

  this->_peerConnection = this->_peerConnectionFactory->CreatePeerConnection(config, &constraints, NULL, NULL, this);

  if (!this->_peerConnection.get())
  {
      std::cerr << "Failed to initialize PeerConnection" << std::endl;
      throw new std::runtime_error("Cannot initialize PeerConnection");
  }


 auto capturer = new Capturer();

 CapturerThread *capturerThread = new CapturerThread(capturer); // This thread sole functionis to call CaptureFrameperiodically


 rtc::scoped_refptr<webrtc::VideoTrackSourceInterface> videoSource = this->_peerConnectionFactory->CreateVideoSource(capturer, NULL);
 rtc::scoped_refptr<webrtc::VideoTrackInterface> videoTrack(this->_peerConnectionFactory->CreateVideoTrack("video_label", videoSource));
 rtc::scoped_refptr<webrtc::MediaStreamInterface> stream = this->_peerConnectionFactory->CreateLocalMediaStream("stream_label");  

 stream->AddTrack(videoTrack);
 if (!this->_peerConnection->AddStream(stream))
 {
    std::cerr << "Adding stream to PeerConnection failed" << std::endl;
    throw new std::runtime_error("Cannot add stream");
 }

 typedef std::pair<std::string,rtc::scoped_refptr<webrtc::MediaStreamInterface>> MediaStreamPair;

 this->_activeStreams.insert(MediaStreamPair(stream->label(), stream));
 webrtc::SdpParseError error;

 webrtc::SessionDescriptionInterface* sessionDescription(webrtc::CreateSessionDescription("offer", this->_offer, &error));

 if (!sessionDescription)
 {
    std::cerr << "Cannot initialize session description." << std::endl;
    throw new std::runtime_error("Cannot set session description");
 }


 this->_peerConnection->SetRemoteDescription(DummySetSessionDescriptionObserver::Create(), sessionDescription);

 this->_peerConnection->CreateAnswer(this, NULL);
}

我的问题是，即使捕获器和对等连接都如预期的那样就绪，也没有视频帧被传输到浏览器客户端。我是不是漏掉了什么？

Answer 1

我把这件事搞清楚了。在创建答案后，根本没有设置本地描述。

调用createAnswer有望触发CreateSessionDescriptionObserver中的OnSuccess回调(在我的例子中是Conductor)。

在那里，我未能将答案保存为本地描述。下面是应该如何实现OnSuccess方法：

void Conductor::OnSuccess(webrtc::SessionDescriptionInterface *desc)
{    
  this->_peerConnection->SetLocalDescription(DummySetSessionDescriptionObserver::Create(), desc);
}

这样做最终会触发参数为kIceGatheringComplete的OnIceGatheringChange，这意味着两端都准备好了。

Answer 2

根据谷歌小组讨论-webrtc thread

您应该实现VideoTrackSourceInterface。在我的例子中，我使用了基类AdaptedVideoTrackSource并创建了一个方法FrameCaptured，它是从我的线程中调用的。在方法FrameCaptured中，我调用了基本方法OnFrame。它工作得很好！

 class StreamSource : public rtc::AdaptedVideoTrackSource
 {
   void OnFrameCaptured(const webrtc::VideoFrame& frame);
 }

 void StreamSource::OnFrameCaptured(const webrtc::VideoFrame& frame) 
 {
   OnFrame(frame);
 }