每第二帧就会下降一次。

Question

我正在运行亲和SDK4.0上的GoPro视频记录。我在Ubuntu16.04上使用一个C++程序。GoPro录像记录60 fps。问题是，Affectiva只为大约一半的帧(即30 fps)提供了结果。如果我看一下Affectiva提供的时间戳，最后一个时间戳与视频持续时间相匹配，这意味着Affectiva以某种方式跳过了第二个帧。

在运行Affectiva之前，我使用以下命令运行ffmpeg，以确保视频的固定帧速率为60 fps：

ffmpeg -i in.MP4 -vf -y -vcodec libx264 -preset medium -r 60 -map_metadata 0:g -strict -2 out.MP4 </dev/null 2>&1

当我使用ffprobe -show_entries frame=pict_type,pkt_pts_time -of csv -select_streams v in.MP4查看演示文稿时间戳时，我得到的原始视频值如下：

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/media/GoPro_concat/GoPro_concat.MP4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
  Duration: 01:14:46.75, start: 0.000000, bitrate: 15123 kb/s
    Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc, bt709), 1280x720 [SAR 1:1 DAR 16:9], 14983 kb/s, 59.94 fps, 59.94 tbr, 60k tbn, 119.88 tbc (default)
    Metadata:
      handler_name    :  GoPro AVC
      timecode        : 13:17:26:44
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      handler_name    :  GoPro AAC
    Stream #0:2(eng): Data: none (tmcd / 0x64636D74)
    Metadata:
      handler_name    :  GoPro AVC
      timecode        : 13:17:26:44
Unsupported codec with id 0 for input stream 2
frame,0.000000,I
frame,0.016683,P
frame,0.033367,P
frame,0.050050,P
frame,0.066733,P
frame,0.083417,P
frame,0.100100,P
frame,0.116783,P
frame,0.133467,I
frame,0.150150,P
frame,0.166833,P
frame,0.183517,P
frame,0.200200,P
frame,0.216883,P
frame,0.233567,P
frame,0.250250,P
frame,0.266933,I
frame,0.283617,P
frame,0.300300,P
frame,0.316983,P
frame,0.333667,P
frame,0.350350,P
frame,0.367033,P
frame,0.383717,P
frame,0.400400,I
frame,0.417083,P
frame,0.433767,P
frame,0.450450,P
frame,0.467133,P
frame,0.483817,P
frame,0.500500,P
frame,0.517183,P
frame,0.533867,I
frame,0.550550,P
frame,0.567233,P
frame,0.583917,P
frame,0.600600,P
frame,0.617283,P
frame,0.633967,P
frame,0.650650,P
frame,0.667333,I
frame,0.684017,P
frame,0.700700,P
frame,0.717383,P
frame,0.734067,P
frame,0.750750,P
frame,0.767433,P
frame,0.784117,P
frame,0.800800,I
frame,0.817483,P
frame,0.834167,P
frame,0.850850,P
frame,0.867533,P
frame,0.884217,P
frame,0.900900,P
frame,0.917583,P
frame,0.934267,I
frame,0.950950,P
frame,0.967633,P
frame,0.984317,P
frame,1.001000,P
frame,1.017683,P
frame,1.034367,P
frame,1.051050,P
frame,1.067733,I
...

我已经在OneDrive上上传了完整的输出。

如果我在原始视频上运行Affectiva (不是由ffmpeg处理)，我将面临同样的问题:丢弃帧。我用的是affdex::VideoDetector detector(60);

ffmpeg命令或Affectiva有问题吗？

编辑：我想我已经知道问题出在哪里了。它似乎是不处理整个视频，但只是停止后，一定数量的处理帧，没有任何错误信息。下面我发布了我正在使用的C++代码。在onProcessingFinished()方法中，当处理完成时，我会将某些内容打印到控制台。但是这条信息从来没有被打印出来，所以“爱”永远不会结束。

我的代码有什么问题吗?还是应该将视频编码成MP4以外的其他格式？

#include "VideoDetector.h"
#include "FrameDetector.h"

#include <iostream>
#include <fstream>
#include <mutex>
#include <condition_variable>

std::mutex m;
std::condition_variable conditional_variable;
bool processed = false;

class Listener : public affdex::ImageListener {
public:
    Listener(std::ofstream * fout) {
        this->fout = fout;
  }
  virtual void onImageCapture(affdex::Frame image){
      //std::cout << "called";
  }
  virtual void onImageResults(std::map<affdex::FaceId, affdex::Face> faces, affdex::Frame image){
      //std::cout << faces.size() << " faces detected:" << std::endl;

      for(auto& kv : faces){
        (*this->fout) << image.getTimestamp() << ",";
        (*this->fout) << kv.first << ",";
        (*this->fout) << kv.second.emotions.joy << ",";
        (*this->fout) << kv.second.emotions.fear << ",";
        (*this->fout) << kv.second.emotions.disgust << ",";
        (*this->fout) << kv.second.emotions.sadness << ",";
        (*this->fout) << kv.second.emotions.anger << ",";
        (*this->fout) << kv.second.emotions.surprise << ",";
        (*this->fout) << kv.second.emotions.contempt << ",";
        (*this->fout) << kv.second.emotions.valence << ",";
        (*this->fout) << kv.second.emotions.engagement << ",";
        (*this->fout) << kv.second.measurements.orientation.pitch << ",";
        (*this->fout) << kv.second.measurements.orientation.yaw << ",";
        (*this->fout) << kv.second.measurements.orientation.roll << ",";
        (*this->fout) << kv.second.faceQuality.brightness << std::endl;


        //std::cout <<  kv.second.emotions.fear << std::endl;
        //std::cout <<  kv.second.emotions.surprise  << std::endl;
        //std::cout <<  (int) kv.second.emojis.dominantEmoji;
      }
  }
private:
    std::ofstream * fout;
};

class ProcessListener : public affdex::ProcessStatusListener{
public:
    virtual void onProcessingException (affdex::AffdexException ex){
        std::cerr << "[Error] " << ex.getExceptionMessage();
    }
    virtual void onProcessingFinished (){
        {
            std::lock_guard<std::mutex> lk(m);
            processed = true;
            std::cout << "[Affectiva] Video processing finised." << std::endl;
        }
        conditional_variable.notify_one();
    }
};

int main(int argc, char ** argsv)
{
    affdex::VideoDetector detector(60, 1, affdex::FaceDetectorMode::SMALL_FACES);
    //affdex::VideoDetector detector(60, 1, affdex::FaceDetectorMode::LARGE_FACES);
    std::string classifierPath="/home/wrafael/affdex-sdk/data";
    detector.setClassifierPath(classifierPath);
    detector.setDetectAllEmotions(true);

    // Output
    std::ofstream fout(argsv[2]);
    fout << "timestamp" << ",";
    fout << "faceId" << ",";
    fout << "joy" << ",";
    fout << "fear" << ",";
    fout << "disgust" << ",";
    fout << "sadness" << ",";
    fout << "anger" << ",";
    fout << "surprise" << ",";
    fout << "contempt" << ",";
    fout << "valence" << ",";
    fout << "engagement"  << ",";
    fout << "pitch" << ",";
    fout << "yaw" << ",";
    fout << "roll" << ",";
    fout << "brightness" << std::endl;

    Listener l(&fout);
    ProcessListener pl;
    detector.setImageListener(&l);
    detector.setProcessStatusListener(&pl);

    detector.start();
    detector.process(argsv[1]);

    // wait for the worker
    {
    std::unique_lock<std::mutex> lk(m);
    conditional_variable.wait(lk, []{return processed;});
    }
    fout.flush();
    fout.close();
}

编辑2：我现在深入研究了这个问题，只查看了一个持续时间为19分钟53的GoPro文件(GoPro拆分了录音)。当我在原始视频上使用affdex::VideoDetector detector(60, 1, affdex::FaceDetectorMode::SMALL_FACES);运行Affectiva时，将生成以下文件。无任何错误信息，也没有打印“确认视频处理”。

当我现在使用ffmpeg -i raw.MP4 -y -vcodec libx264 -preset medium -r 60 -map_metadata 0:g -strict -2 out.MP4转换视频，然后用affdex::VideoDetector detector(60, 1, affdex::FaceDetectorMode::SMALL_FACES);运行Affectiva时，Affectiva会一直运行到结束，并打印"Affectiva视频处理细化“，但是帧速率仅为23 fps。这里是文件。

现在，当我在这个转换的文件上使用affdex::VideoDetector detector(62, 1, affdex::FaceDetectorMode::SMALL_FACES);运行Affectiva时，将会在509 s之后停止，并且没有打印"Affectiva视频处理细化“。这里是文件。

Answer 1

如果视频帧速率为60，则使用高于60的数字来处理所有帧。如果你只使用61或62，你应该得到正确的帧数。