在android上显示mpegts流中的h264视频

Question

在android上显示来自mpegts流的h264视频。

几天来，我一直在努力让这件事奏效，但没有成功。我所拥有的是一个产生h264视频流的设备，它通过原始udp (而不是rtp)在mpegts容器中多播。我正试着让它在android上的定制应用程序中显示。

我读到在MediaPlayer中内置的android支持h264和mpegts，但是它不处理udp:// streams，所以我不能使用它(这将是目前为止最简单的)。相反，我尝试将mpegts流手动解析为基本流，并将其传递给已传递给MediaCodec的SurfaceView表面。无论我似乎在尝试什么，都会发生两件事(一旦我修正了异常，等等)：

• SurfaceView总是黑色的。
• MediaCodec总是接受大约6-9个缓冲区，然后dequeueInputBuffer就会立即开始失败(返回-1)，我不能再排队了。

我可以将mpeg流分割成TS数据包，然后将它们的有效负载加入到PES数据包中。我尝试将完整的PES数据包(减去PES头)传递到MediaCodec中。

我还尝试通过在\x00\x00\x01上拆分并将它们分别传递到MediaCodec中，将PES数据包拆分成单独的NAL单元。

我也尝试过推迟在NAL单元中传递，直到我收到了SPS单元，并且首先用BUFFER_FLAG_CODEC_CONFIG传递了它。

所有这些都导致了上面提到的同样的事情。我不知道该尝试什么，所以任何帮助都是非常感谢的。

有几点我还不太清楚：

• 几乎我看到的所有示例都是从MediaFormat中获得的，我无法在流中使用它。少数不使用MediaExtractor解释的字节字符串中的csd-0和cs-1。我读到SPS数据包可以放在缓冲区中，所以这就是我所尝试的。
• 我不知道该把什么传递给演示时间。TS包有一个PCR，PES包有一个PTS，但是我不知道api所期望的是什么以及它们之间的关系。
• 我不确定如何将数据传递到MediaCodec (这就是为什么它不再为我提供缓冲区了吗？)我从这样的帖子中得到了通过单个NAL单元的想法：在android中解码原始H264流？

我用来做这个例子的其他参考资料：

• MPEG-TS格式
• PES格式
• PES格式

代码(抱歉太长了)：

我刚刚用AndroidStudio中的基本模板创建了一个测试应用程序，其中大部分都是样板，所以我只会粘贴视频相关的内容。

SurfaceView是在xml中定义的，所以当它被创建/更改时，抓住它并得到它的表面。

public class VideoPlayer extends Activity implements SurfaceHolder.Callback {
    private static final String TAG = VideoPlayer.class.getName();

    PlayerThread playerThread;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_video_player);

        SurfaceView view = (SurfaceView) findViewById(R.id.surface);
        view.getHolder().addCallback(this);

    }

    ...

    @Override
    public void surfaceCreated(SurfaceHolder surfaceHolder) {
        Log.d(TAG,"surfaceCreated");
    }

    @Override
    public void surfaceChanged(SurfaceHolder surfaceHolder, int i, int i2, int i3) {
        Log.d("main","surfaceChanged");
        if( playerThread == null ) {
            playerThread = new PlayerThread(surfaceHolder.getSurface());
            playerThread.start();
        }
    }

    ...

PlayerThread是一个内部类，它从多播端口读取数据并将其传递给后台线程上的解析函数：

class PlayerThread extends Thread {
    private final String TAG = PlayerThread.class.getName();

    MediaExtractor extractor;
    MediaCodec decoder;
    Surface surface;
    boolean running;

    ByteBuffer[] inputBuffers;

    public PlayerThread(Surface surface)
    {
        this.surface = surface;

        MediaFormat format = MediaFormat.createVideoFormat("video/avc",720,480);

        decoder = MediaCodec.createDecoderByType("video/avc");
        decoder.configure(format, surface, null, 0);
        decoder.start();

        inputBuffers = decoder.getInputBuffers();

    }

    ...

    @Override
    public void run() {
        running = true;
        try {

            String mcg = "239.255.0.1";
            MulticastSocket ms;

            ms = new MulticastSocket(1841);
            ms.joinGroup(new InetSocketAddress(mcg, 1841), NetworkInterface.getByName("eth0"));
            ms.setSoTimeout(4000);
            ms.setReuseAddress(true);

            byte[] buffer = new byte[65535];
            DatagramPacket dp = new DatagramPacket(buffer, buffer.length);

            while (running) {
                try {
                    ms.receive(dp);
                    parse(dp.getData());

                } catch (SocketTimeoutException e) {
                    Log.d("thread", "timeout");
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

接收工作正常，每个数据包包含两个TS包。它们被传递给解析函数：

    boolean first = true;
    ByteArrayOutputStream current =  new ByteArrayOutputStream();
    void parse(byte[] data) {
        ByteBuffer stream = ByteBuffer.wrap(data);
        // mpeg-ts stream header is 4 bytes starting with the sync byte
        if( stream.get(0) != 0x47 ) {
            Log.w(TAG, "got packet w/out mpegts header!");
            return;
        }

        ByteBuffer raw = stream.duplicate();
        // ts packets are 188 bytes
        raw.limit(188);
        TSPacket ts = new TSPacket(raw);
        if( ts.pid == 0x10 ) {
            processTS(ts);
        }

        // move to second packet
        stream.position(188);
        stream.limit(188*2);
        if( stream.get(stream.position()) != 0x47 ) {
            Log.w(TAG, "missing mpegts header!");
            return;
        }
        raw = stream.duplicate();
        raw.limit(188*2);
        ts = new TSPacket(raw);
        if( ts.pid == 0x10 ) {
            processTS(ts);
        }
    }

TS数据包由TSPacket类解析：

public class TSPacket {
    private final static String TAG = TSPacket.class.getName();

    class AdaptationField {

        boolean di;
        boolean rai;
        boolean espi;
        boolean hasPcr;
        boolean hasOpcr;
        boolean spf;
        boolean tpdf;
        boolean hasExtension;

        byte[] data;

        public AdaptationField(ByteBuffer raw) {
            // first byte is size of field minus size byte
            int count = raw.get() & 0xff;

            // second byte is flags
            BitSet flags = BitSet.valueOf(new byte[]{ raw.get()});

            di = flags.get(7);
            rai = flags.get(6);
            espi = flags.get(5);
            hasPcr = flags.get(4);
            hasOpcr = flags.get(3);
            spf = flags.get(2);
            tpdf = flags.get(1);
            hasExtension = flags.get(0);

            // the rest is 'data'
            if( count > 1 ) {
                data = new byte[count-1];
                raw.get(data);
            }
        }
    }

    boolean tei;
    boolean pus;
    boolean tp;
    int pid;
    boolean hasAdapt;
    boolean hasPayload;
    int counter;
    AdaptationField adaptationField;
    byte[] payload;

    public TSPacket(ByteBuffer raw) {
        // check for sync byte
        if( raw.get() != 0x47 ) {
            Log.e(TAG, "missing sync byte");
            throw new InvalidParameterException("missing sync byte");
        }

        // next 3 bits are flags
        byte b = raw.get();
        BitSet flags = BitSet.valueOf(new byte[] {b});

        tei = flags.get(7);
        pus = flags.get(6);
        tp = flags.get(5);

        // then 13 bits for pid
        pid = ((b << 8) | (raw.get() & 0xff) ) & 0x1fff;

        b = raw.get();
        flags = BitSet.valueOf(new byte[]{b});

        // then 4 more flags
        if( flags.get(7) || flags.get(6) ) {
            Log.e(TAG, "scrambled?!?!");
            // todo: bail on this packet?
        }

        hasAdapt = flags.get(5);
        hasPayload = flags.get(4);

        // counter
        counter = b & 0x0f;

        // optional adaptation field
        if( hasAdapt ) {
            adaptationField = new AdaptationField(raw);
        }

        // optional payload field
        if( hasPayload ) {
            payload = new byte[raw.remaining()];
            raw.get(payload);
        }
    }

}

然后传递给processTS函数：

    // a PES packet can span multiple TS packets, so we keep track of the 'current' one
    PESPacket currentPES;
    void processTS(TSPacket ts) {
        // payload unit start?
        if( ts.pus ) {
            if( currentPES != null ) {
                Log.d(TAG,String.format("replacing pes: len=%d,size=%d", currentPES.length, currentPES.data.size()));
            }
            // start of new PES packet
            currentPES = new PESPacket(ts);
        } else if (currentPES != null ) {
            // continued PES
            currentPES.Add(ts);
        } else {
            // haven't got a start pes yet
            return;
        }

        if( currentPES.isFull() ) {
            long pts = currentPES.getPts();
            byte[] data = currentPES.data.toByteArray();

            int idx = 0;

            do {
                int sidx = idx;
                // find next NAL prefix
                idx = Utility.indexOf(data, sidx+4, data.length-(sidx+4), new byte[]{0,0,1});

                byte[] NAL;
                if( idx >= 0 ) {
                    NAL = Arrays.copyOfRange(data, sidx, idx);
                } else {
                    NAL = Arrays.copyOfRange(data, sidx, data.length);
                }

                // send SPS NAL before anything else
                if( first ) {
                    byte type = NAL[3] == 0 ? NAL[4] : NAL[3];
                    if( (type & 0x1f) == 7 ) {
                        Log.d(TAG, "found sps!");

                        int ibs = decoder.dequeueInputBuffer(1000);
                        if (ibs >= 0) {
                            ByteBuffer sinput = inputBuffers[ibs];
                            sinput.clear();
                            sinput.put(NAL);

                            decoder.queueInputBuffer(ibs, 0, NAL.length, 0, MediaCodec.BUFFER_FLAG_CODEC_CONFIG);
                            Log.d(TAG, "sent sps");
                            first = false;
                        } else
                            Log.d(TAG, String.format("could not send sps! %d", ibs));
                    }
                } else {

                    // put in decoder?
                    int ibs = decoder.dequeueInputBuffer(1000);
                    if (ibs >= 0) {
                        ByteBuffer sinput = inputBuffers[ibs];
                        sinput.clear();
                        sinput.put(NAL);

                        decoder.queueInputBuffer(ibs, 0, NAL.length, 0, 0);
                        Log.d(TAG, "buffa");
                    } 
                }
            } while( idx >= 0 );

            // finished with this pes
            currentPES = null;
        }
    }

PES数据包由PESPacket类解析：

public class PESPacket {
    private final static String TAG = PESPacket.class.getName();

    int id;
    int length;

    boolean priority;
    boolean dai;
    boolean copyright;
    boolean origOrCopy;
    boolean hasPts;
    boolean hasDts;
    boolean hasEscr;
    boolean hasEsRate;
    boolean dsmtmf;
    boolean acif;
    boolean hasCrc;
    boolean pesef;
    int headerDataLength;

    byte[] headerData;
    ByteArrayOutputStream data = new ByteArrayOutputStream();

    public PESPacket(TSPacket ts) {
        if( ts == null || !ts.pus) {
            Log.e(TAG, "invalid ts passed in");
            throw new InvalidParameterException("invalid ts passed in");
        }

        ByteBuffer pes = ByteBuffer.wrap(ts.payload);

        // start code
        if( pes.get() != 0 || pes.get() != 0 || pes.get() != 1 ) {
            Log.e(TAG, "invalid start code");
            throw new InvalidParameterException("invalid start code");
        }

        // stream id
        id = pes.get() & 0xff;

        // packet length
        length = pes.getShort() & 0xffff;

        // this is supposedly allowed for video
        if( length == 0 ) {
            Log.w(TAG, "got zero-length PES?");
        }

        if( id != 0xe0 ) {
            Log.e(TAG, String.format("unexpected stream id: 0x%x", id));
            // todo: ?
        }

        // for 0xe0 there is an extension header starting with 2 bits '10'
        byte b = pes.get();
        if( (b & 0x30) != 0 ) {
            Log.w(TAG, "scrambled ?!?!");
            // todo: ?
        }

        BitSet flags = BitSet.valueOf(new byte[]{b});
        priority = flags.get(3);
        dai = flags.get(2);
        copyright = flags.get(1);
        origOrCopy = flags.get(0);

        flags = BitSet.valueOf(new byte[]{pes.get()});
        hasPts = flags.get(7);
        hasDts = flags.get(6);
        hasEscr = flags.get(5);
        hasEsRate = flags.get(4);
        dsmtmf = flags.get(3);
        acif = flags.get(2);
        hasCrc = flags.get(1);
        pesef = flags.get(0);

        headerDataLength = pes.get() & 0xff;

        if( headerDataLength > 0 ) {
            headerData = new byte[headerDataLength];
            pes.get(headerData);
        }

        WritableByteChannel channel = Channels.newChannel(data);
        try {
            channel.write(pes);
        } catch (IOException e) {
            e.printStackTrace();
        }

        // length includes optional pes header,
        length = length - (3 + headerDataLength);
    }

    public void Add(TSPacket ts) {
        if( ts.pus ) {
            Log.e(TAG, "don't add start of PES packet to another packet");
            throw new InvalidParameterException("ts packet marked as new pes");
        }

        int size = data.size();
        int len = length - size;
        len = ts.payload.length > len ? len : ts.payload.length;
        data.write(ts.payload, 0, len);
    }

    public boolean isFull() {
        return (data.size() >= length );
    }

    public long getPts() {
        if( !hasPts || headerDataLength < 5 )
            return 0;

        ByteBuffer hd = ByteBuffer.wrap(headerData);
        long pts = ( ((hd.get() & 0x0e) << 29)
                    | ((hd.get() & 0xff) << 22)
                    | ((hd.get() & 0xfe) << 14)
                    | ((hd.get() & 0xff) << 7)
                    | ((hd.get() & 0xfe) >>> 1));

        return pts;
    }
}

Answer 1

因此，我最终发现，即使我使用的是输出面，我也必须手动排出输出缓冲区。通过调用decoder.dequeueOutputBuffer，然后调用decoder.releaseOutputBuffer，输入缓冲区按预期工作。

我还能够通过传入单独的NAL单元和全接入单元(每个PES包一个)获得输出，但我通过传入全接入单元获得了最清晰的视频。