Video Decoder

The Video Decoder HAL service provides interfaces for passing compressed video to the vendor layer for decoding. If the service supports secure audio processing, it may also handle secure buffers.

The output of the video decoder can follow two paths:

Non-tunnelled mode – The decoded video is returned to the RDK media pipeline as a video frame buffer along with metadata.
Tunnelled mode – The decoded video is passed directly through the vendor layer.

The choice between tunnelled and non-tunnelled video does not affect the operational mode of the audio decoder. It is possible to have tunnelled video while using non-tunnelled audio.

The video decoder's operational mode can be selected by the client upon initialization.

The RDK middleware GStreamer pipeline includes a dedicated RDK Video Decoder element, specifically designed to integrate with the Video Decoder HAL interface.

References

Info


Interface Definition	videodecoder/current
Interface Version	`current`
API Documentation	TBD
HAL Interface Type	AIDL and Binder
VTS Tests	TBC
Reference Implementation - vComponent	https://github.com/rdkcentral/rdk-halif-aidl/tree/main/videodecoder/current

Related Pages

Implementation Requirements

#	Requirement	Comments
HAL.VIDEODECODER.1	Video elementary stream data shall be passed to the video decoder to be decoded as frames.	Decontainerisation and demuxing are performed by the RDK AV pipeline before the video decoder.
HAL.VIDEODECODER.2	Encoded video data is passed one frame at a time to the video decoder in stream, file or broadcast delivered order.
HAL.VIDEODECODER.3	Decoded video frames are output from the decoder in presentation order.
HAL.VIDEODECODER.4	Only one video frame shall be output per output frame callback from the video decoder.
HAL.VIDEODECODER.5	Encoded video data shall be passed in shared memory buffers by handle and shall be in either secure or non-secure buffer types.	The pool implementation of memory buffers for secure and non-secure memory is implemented by the vendor. See AV Buffer for details.
HAL.VIDEODECODER.6	The video decoder shall support a secure video pipeline where encoded and decoded data in secure buffers shall not be exposed to any process outside of the secure video pipeline. Secure coded video input buffers to the video decoder shall always be output in secure decoded frame buffers.
HAL.VIDEODECODER.7	The video decoder shall operate in either a tunnelled or non-tunnelled operational mode.	Only one of these operational modes needs be supported by the video decoder.
HAL.VIDEODECODER.8	The video decoder may optionally operate in a textured video operational mode.
HAL.VIDEODECODER.9	The video decoder HAL shall report on the number of video decoder instances supported and their capabilities.
HAL.VIDEODECODER.10	An opened video decoder instance shall be configured to decode only a single codec type.	No dynamic video codec switching is supported while open. A video decoder instance must be closed and reopened to change the codec type.
HAL.VIDEODECODER.11	The video decoder shall be able to decode back to back I-frames.	Used in I-frame trick modes.
HAL.VIDEODECODER.12	The video decoder shall discard any frames until the first reference frame has been received.
HAL.VIDEODECODER.13	If a client process exits, the Video Decoder server shall automatically stop and close any Video Decoder instance controlled by that client.

Interface Definition

Interface Definition File	Description
`IVideoDecoderManager.aidl`	Video Decoder Manager HAL which provides access to IVideoDecoder resource instances.
`IVideoDecoder.aidl`	Video Decoder interface for a single video decoder resource instance.
`IVideoDecoderController.aidl`	Controller interface for an IVideoDecoder resource instance.
`IVideoDecoderControllerListener.aidl`	Listener callbacks interface to clients from an IVideoDecoderController.
`IVideoDecoderEventListener.aidl`	Listener callbacks interface to clients from an IVideoDecoder.
`Capabilities.aidl`	Parcelable describing the capabilities of an IVideoDecoder resource instance.
`Codec.aidl`	Enum list of video codecs.
`CodecCapabilities.aidl`	Parcelable describing the capabilities of a codec supported by an IVideoDecoder.
`CSDVideoFormat.aidl`	Enum list of video codec specific data formats.
`DynamicRange.aidl`	Enum list of dynamic ranges.
`ErrorCode.aidl`	Enum list of video decoder error codes.
`FrameMetadata.aidl`	Parcelable of video frame metadata passed from the video decoder.
`OperationalMode.aidl`	Enum list of video decoder operational modes.
`PixelFormat.aidl`	Enum list of video pixel formats.
`Property.aidl`	Enum list of video decoder properties.
`PropertyKVPair.aidl`	Parcelable of a Property and PropertyValue pair.
`ScanType.aidl`	Enum list of video frame scan types.

Initialization

The systemd hal-video_decoder_manager.service unit file is provided by the vendor layer to start the service and should include Wants or Requires directives to start any platform driver services it depends upon.

The Video Decoder Manager service depends on the Service Manager to register itself as a service.

Upon starting, the service shall register the IVideoDecoderManager interface with the Service Manager using the String IVideoDecoderManager.serviceName and immediately become operational.

Product Customization

The IVideoDecoderManager.getVideoDecoderIds() should return an array of IVideoDecoder.Id parcelables to uniquely represent all of the video decoder resources supported by the vendor layer. Typically, the ID value starts at 0 for the first video decoder and increments by 1 for each additional video decoder.

The Capabilities parcelable returned by the IVideoDecoder.getCapabilities() function lists all of the Codec types and DynamicRange types supported by this video decoder instance and indicates if the secure audio path can be used.

Each entry in supportedCodecs[] (a CodecCapabilities parcelable) declares an explicit supportedOutputPixelFormats[] list — the chroma formats the decoder hardware can produce on its output for that codec/profile/level. The middleware uses this at capability-discovery time to reject incompatible streams before binding the decoder, rather than failing inside decodeBufferWithMetadata() after start. Hardware support typically varies per codec block (e.g. a SoC's VP9 path may produce 4:4:4 on the same silicon where the H.264 path is 4:2:0 only), so the list is per-codec entry rather than a single global list.

A video decoder instance may support any number of video codecs, but can only operate on one compressed video stream in an open session. Concurrent video decode requires multiple video decoder instances to be opened.

System Context

The Video Decoder HAL can provide functionality to multiple clients.

Typically an RDK middleware GStreamer video decoder element will work with a single IVideoDecoder instance and pass it AV Buffer handles for decode.

The RDK middleware resource management system will examine the number of video decoder resources and their capabilities, so they can be allocated to streaming sessions.

flowchart TD
    RDKClientComponent("RDKClientComponent")
    subgraph Listeners["Listeners"]
        IVideoDecoderEventListener("IVideoDecoderEventListener")
        IVideoDecoderControllerListener("IVideoDecoderControllerListener")
    end
    subgraph IVideoDecoderHAL["Video Decoder HAL"]
        IVideoDecoderManager("IVideoDecoderManager <br>(Service)")
        IVideoDecoder("IVideoDecoder <br>(Instance)")
        IVideoDecoderController("IVideoDecoderController <br>(Instance)")
    end
    subgraph OutputComponents["Output"]
        VideoFramePool("Video Frame Pool")
        VideoFrameQueue["Video Frame Queue"]
    end
    RDKClientComponent -- createVideoPool() <br> alloc() <br> free() <br> destroyPool() --> IAVBuffer
    RDKClientComponent -- getVideoDecoderIds() <br> getVideoDecoder() getSupportedOperationModes()--> IVideoDecoderManager
    RDKClientComponent -- getCapabilities() <br> getProperty() <br> getPropertyMulti() <br> getState() <br> open() <br> close() <br> registerEventListener() <br> unregisterEventListener()--> IVideoDecoder
    RDKClientComponent -- registerEventListener() <br> unregisterEventListener() --> IVideoDecoder
    RDKClientComponent -- start() <br> stop() <br> setProperty() <br> decodeBufferWithMetadata() <br> flush() <br> signalDiscontinuity() <br> parseCodecSpecificData() --> IVideoDecoderController
    IVideoDecoderManager --> IVideoDecoder --> IVideoDecoderController
    IVideoDecoder -- onStateChanged() <br> onDecodeError() --> IVideoDecoderEventListener
    IVideoDecoderEventListener --> RDKClientComponent
    IVideoDecoderControllerListener --> RDKClientComponent
    IVideoDecoderController -- onFrameOutput() --> IVideoDecoderControllerListener
    IVideoDecoderController -- onUserDataOutput() --> IVideoDecoderControllerListener
    IVideoDecoderController -- alloc --> VideoFramePool
    IVideoDecoderManager -- free --> IAVBuffer
    IVideoDecoderController -- tunneled Video --> VideoFrameQueue

    classDef background fill:#121212,stroke:none,color:#E0E0E0;
    classDef blue fill:#1565C0,stroke:#E0E0E0,stroke-width:2px,color:#E0E0E0;
    classDef lightGrey fill:#616161,stroke:#E0E0E0,stroke-width:2px,color:#FFFFFF;
    classDef wheat fill:#FFB74D,stroke:#424242,stroke-width:2px,color:#000000;
    classDef green fill:#4CAF50,stroke:#E0E0E0,stroke-width:2px,color:#FFFFFF;
    classDef default fill:#1E1E1E,stroke:#E0E0E0,stroke-width:1px,color:#E0E0E0;

    RDKClientComponent:::blue
    IVideoDecoderManager:::wheat
    IVideoDecoderController:::wheat
    IVideoDecoder:::wheat
    IAVBuffer:::green
    IVideoDecoderControllerListener:::wheat
    IVideoDecoderEventListener:::wheat
    VideoFramePool:::green
    VideoFrameQueue:::green

Resource Management

The IVideoDecoderManager provides access to one or more IVideoDecoder sub-interfaces which each represent a video decoder resource instance offered by the platform.

Each IVideoDecoder resource instance is assigned a unique integer ID, which is used in IVideoDecoder.Id.value and can be read from RESOURCE_ID using the IVideoDecoder.getProperty() function.

To use an IVideoDecoder resource instance it must be opened by a client, which returns an IVideoDecoderController sub-interface to access buffer decoding and additional state controls.

Important

Any number of clients can access the IVideoDecoderManager service and get access to the IVideoDecoder sub-interfaces, but only 1 client can open() an IVideoDecoder and access its IVideoDecoderController sub-interface.

The diagram below shows the relationship between the interfaces and resource instances.

graph LR

    %% --- Encapsulating Everything Inside "Video Decoder HAL" ---
    IVideoDecoderManager("IVideoDecoderManager")

    %% --- Video Decoder Manager Service Spawns Instances ---
    IVideoDecoderManager --> ADI1("IVideoDecoder <br> ID = 0")
    IVideoDecoderManager --> ADI2("IVideoDecoder <br> ID = 1")
    IVideoDecoderManager --> ADI3("IVideoDecoder <br> ID = 2")

    %% --- Each Instance Has a Controller ---
    ADI1 --> ADIC1("IVideoDecoderController")
    ADI2 --> ADIC2("IVideoDecoderController")
    ADI3 --> ADIC3("IVideoDecoderController")

    %% --- High Contrast Styling (Rounded Box Simulation) ---
    classDef background fill:#121212,stroke:none,color:#E0E0E0;
    classDef manager fill:#388E3C,stroke:#1B5E20,stroke-width:2px,color:#FFFFFF;
    classDef instance1 fill:#FFC107,stroke:#FF8F00,stroke-width:2px,color:#000000;
    classDef instance2 fill:#FF9800,stroke:#E65100,stroke-width:2px,color:#000000;
    classDef instance3 fill:#F44336,stroke:#B71C1C,stroke-width:2px,color:#FFFFFF;
    classDef controller fill:#00ACC1,stroke:#006064,stroke-width:2px,color:#000000;

    %% --- Apply Colors ---
    class IVideoDecoderManager manager;
    class ADI1 instance1;
    class ADI2 instance2;
    class ADI3 instance3;
    class ADIC1 instance1;
    class ADIC2 instance2;
    class ADIC3 instance3;

    %% --- Consistent Link Colors Per Instance ---
    %% Yellow for Instance 0
    linkStyle 0,3 stroke:#AA8800,stroke-width:2px;
    %% Orange for Instance 1
    linkStyle 1,4 stroke:#CC5500,stroke-width:2px;
    %% Red for Instance 2
    linkStyle 2,5 stroke:#CC2200,stroke-width:2px;

Codec Support

Each RDK product defines the video codecs it requires for IP streaming, apps and broadcast.

Some codecs are subject to third party licensing agreements and may therefore only be included in some products.

The list below indicates the list of video codecs which are mandatory for the platform to support as hardware/vendor decode (potentially secure).

Codec	Typical Use Case	Hardware/Vendor Support
MPEG-2 H.262 up to 1080P@60fps	Broadcast streams, IP streams, files	Mandatory
AVC H.264 upto HP@L5.1 up to 4Kx2K@30fps	Broadcast streams, IP streams, files	Mandatory
HEVC H.265 upto MP-10@L5.1 up to 4Kx2K@60fps	Broadcast streams, IP streams, files	Mandatory
VP9 upto Profile 2-10 up to 4Kx2K@60fps	YouTube, WebM, files	Mandatory
AV1 upto MP-10@L5.1 up to 4Kx2K@60fps	YouTube, WebM, files	Mandatory

Encrypted Video Playback

Encrypted video is copied into a non-secure buffer by the application and then decrypted into a secure buffer. The secure buffer is then decoded by a video decoder accessed through the IVideoDecoderController interface.

Secure Video Processing

Secure video processing (SVP) is a requirement for RDK-E. Video decoder instances shall declare themselves as secure or non-secure by setting Capabilities.supportsSecure appropriately.

A secure video decoder shall be able to handle secure AV buffers and decoded video frames output from the decoder shall be either contained in secure AV buffers or securely tunnelled in the vendor layer.

If any video decoder supports SVP in non-tunnelled mode then the Video Sink HAL must also support SVP to be able to process secure AV buffers of decoded video frames.

Markdown Output:

Operational Modes

There are 3 modes that video decoders can operate in. The IVideoDecoderManager.getSupportedOperationalModes() function must return all operational modes supported by the video decoders in the system.

This set of advertised operational modes must operate on all video decoder instances.

Tunnelled and non-tunnelled modes cannot operate at the same time. It is optional for video decoders to support both modes as options, but at least one of them must be supported.

If both are supported then there shall never be a dynamic switch between the 2 modes while STARTED.

The OPERATIONAL_MODE property controls the operational mode the video decoder shall use.

The enum OperationalMode provides the constants used to specify operational modes and allows for bitwise-or of multiple values.

The video decoder may switch operational modes at any time while in a READY or STARTED state.

In all modes, AV buffers containing compressed video are passed into the video decoder through calls to decodeBufferWithMetadata().

Operational Mode	Description
`TUNNELLED`	Decoded video frames are passed directly to the linked video sink and video plane for rendering. The vendor layer is responsible for AV sync where audio and video streams are linked in the same pipeline. Decoded video frames are never received back in frame buffers over `IVideoDecoderControllerListener.onFrameOutput()`, but the `FrameMetadata` must still be returned in the usual way. All calls to `onFrameOutput()` shall have the `frameBufferHandle` set to -1 to indicate no video frame buffer handle is being passed back. It is optional for video decoders to support tunnelled operational mode. If supported, tunnelled mode may be dynamically enabled or disabled while the video decoder is `READY` or `STARTED` and may run concurrently with graphics texture mode. Tunnelled mode cannot be used at the same time as non-tunnelled mode.
`NON_TUNNELLED`	Decoded video frames are received back in video frame buffers over `IVideoDecoderListener.onFrameOutput()`. Frames must be received in presentation order. It is optional for video decoders to support non-tunnelled operational mode. If supported, non-tunnelled mode may be dynamically enabled or disabled while the video decoder is `READY` or `STARTED` and may run concurrently with graphics texture mode. Non-tunnelled mode cannot be used at the same time as tunnelled mode.
`GRAPHICS_TEXTURE`	Video frames are converted to NV12 textures. It is optional for video decoders to support graphics texture operational mode. If supported, graphics texture mode may be dynamically enabled or disabled while the video decoder is `READY` or `STARTED` and may run concurrently with tunnelled or non-tunnelled mode.

Frame Metadata

As video frames are decoded, the metadata which related to the frames must be passed to the client over IVideoDecoderControllerListener.onFrameOutput().

In non-tunnelled operating mode, the frame buffer handle and metadata related to the frame must be passed in the same onFrameOutput() call.

To conserve CPU load, the frame metadata is only passed with the first decoded frame after a start(), the first decoded frame after a flush() or if the frame metadata changes.

If the frame metadata does not need to be passed, then the @nullable FrameMetadata metadata parameter should be passed as null in onFrameOutput().

The same rules for frame metadata apply to all operational modes.

When operating exclusively in tunnelled mode, if there is no frame metadata to be passed, then no call to onFrameOutput() should be made because there is no frame buffer handle or frame metadata to return to the client.

Stream Hints vs Bitstream Truth

The video decoder accepts a set of stream-description hints before decoding begins, via setters on IVideoDecoderController:

setStreamResolution(width, height) — coded frame dimensions
setFrameRate(numerator, denominator) — frame rate as a rational
setColorimetry(colorimetry) — colour primaries / matrix
setMasteringDisplayInfo(info) — HDR mastering display volume (SMPTE ST 2086)
setContentLightLevel(info) — HDR MaxCLL / MaxFALL (CTA-861.3)
setDolbyVisionLayerFlags(blPresent, elPresent) — DV layer configuration
setPixelAspectRatio(parX, parY) — pixel aspect ratio (e.g. 10:11 for NTSC SD)

These hints are sourced from container or codec-config metadata (MP4 boxes, DASH MPD, HLS playlist, GstCaps from upstream parser elements). They are always available in the clear — even under SVP — because the container and codec-config (avcC / hvcC) are never encrypted.

Precedence — bitstream wins

When the decoder later finds an explicit value in the bitstream (H.264/HEVC VUI parameters, in-stream SEI, DV RPU), the bitstream value takes precedence over the hint. The hint is only used as a fallback when the bitstream is silent on that field.

This matters for legacy or partial streams where the bitstream omits VUI / SEI metadata that the container carries (e.g. SD MPEG-2 with PAR only in pasp, some packagers' H.264 output with no VUI colour_description). In those cases the hint is the only source of the value, and downstream display configuration would otherwise be incorrect.

For hints whose contract is "tell the decoder what to render, and the decoder echoes back what it actually used", the decoder reports the value it actually used — bitstream-derived or hint-derived — in FrameMetadata:

Hint setter	Reported in FrameMetadata as
`setStreamResolution`	`codedWidth`, `codedHeight`
`setFrameRate`	`frameRateNumerator`, `frameRateDenominator`
`setColorimetry`	`colorimetry`
`setMasteringDisplayInfo`	`masteringDisplayInfo`
`setContentLightLevel`	`contentLightLevel`
`setPixelAspectRatio`	`parX`, `parY`

setDolbyVisionLayerFlags is the one exception: it is a decoder-configuration hint (selects single-layer vs dual-layer DV decode mode) and is not mirrored into a FrameMetadata field. The bitstream-precedence rule still applies — DV RPU signalling overrides the hint when present — but the result is observable only as a change in decode behaviour, not as a readback field.

Persistence

Hints describe the stream, not the playback session. They:

Persist across flush() — middleware does not need to re-set them after every flush; the same stream description still applies.
Are cleared on close() — when the decoder instance is closed and re-opened against a different stream, the new caller is responsible for re-seeding the hints from the new container metadata.

State precondition

All hint setters require State::READY. They are setup-time configuration, not runtime reconfiguration. Bitstream-derived changes mid-playback (e.g. ABR rep switch carrying new SPS) flow through the bitstream automatically and surface via FrameMetadata without middleware action.

Behaviour under SVP / encrypted pipeline

Secure Video Path (SVP) makes the hint setters more important, not less. Under SVP the input pipeline splits cleanly between what middleware can read and what only the decoder can read after decryption:

Data	Always clear?	Visible to middleware?	Visible to decoder (after decrypt)?
Container metadata (MP4 `colr`/`mdcv`/`pasp`, DASH MPD, HLS playlist)	Yes	Yes — passed via hint setters	Not directly
Codec config (`avcC` / `hvcC` SPS/PPS array)	Yes (CENC requires)	Yes — `h264parse` / `h265parse` extract VUI fields, also surfaced via hints	Yes — same data passed by middleware on init
Non-VCL NAL units (SPS / PPS / SEI / AUD) in stream	Implementation-defined per packager (typically clear, can be encrypted)	Only if packager left them clear	Yes — after decrypt
VCL NAL units (slices)	Encrypted	No	Yes — after decrypt

Implications:

Hints are the only reliable middleware-visible source of stream description under SVP. The middleware path (demuxer → parser → caps) extracts everything it can from clear container + codec-config data and must push it via the hint setters because the bitstream slices are opaque to it.
Decoder is the only place that has clear access to in-band SEI under SVP (H.264/HEVC SEI payload type 137 "mastering display colour volume", type 144 "content light level information", type 147 "alternative transfer characteristics", DV RPU SEI type 4). For these fields:
If middleware can derive them from the container (e.g. mdcv / cclv MP4 boxes), the hint setter is the path.
If the value only exists in encrypted in-band SEI, the decoder parses internally and reports via FrameMetadata — middleware cannot pre-set them.
Bitstream-precedence still applies. Even under SVP, when the decoder finds a value in the (decrypted) bitstream it overrides the hint and is reported via FrameMetadata.
No information is leaked by the hint flow — hints carry only data the middleware already has from the always-clear container / codec-config; they don't expose anything from the encrypted slice payload.

This is also why the decoder must own a "secure parser" path for any metadata that can only be recovered from encrypted in-band SEI — middleware cannot substitute for it. See discussion #367 for the broader secure-parser design conversation.

Low Latency Mode

A media pipeline is operating in low latency mode when the video decoder and audio decoder (if present) are set with a LOW_LATENCY_MODE property to 1 (enabled).

Video Stream Discontinuities

Where the client has knowledge of PTS discontinuities in the video stream, it shall call IVideoDecoderController.signalDiscontinuity() between the AV buffers passed to decodeBufferWithMetadata().

For the first input AV Buffer video frame passed in for decode after the discontinuity, it shall indicate the discontinuity in its next output FrameMetadata.

End of Stream Signalling

EOS rides entirely on the framework metadata parcelables on both sides of the interface. There is no separate signal method.

Input side: the client sets InputBufferMetadata.endOfStream = true on the final call to IVideoDecoderController.decodeBufferWithMetadata(). bufferHandle MUST reference a valid encoded frame - there is no EOS-only marker form and no path to signal EOS without data. If the client has no more data to send, it ends the session via stop() (or flush(reset=true) if the decoder is to be reused).

Output side: EOS rides on the FINAL IVideoDecoderControllerListener.onFrameOutput() callback of the decode session by FrameMetadata.endOfStream = true. There is no separate EOS-only marker callback after the last frame. Fires exactly once per session. In non-tunnelled mode the callback delivers the last decoded frame with valid frameAVBufferHandle and FrameMetadata; in tunnelled mode frameAVBufferHandle = -1 as normal. metadata is guaranteed non-null on the EOS callback (because endOfStream transitioning from false to true is a metadata change) so clients can rely on metadata != null && metadata.endOfStream for unambiguous EOS detection. The other fields of FrameMetadata describe the final frame as normal.

In-bitstream EOS: for codecs that carry an elementary-stream EOS marker (MPEG-2 sequence_end_code, H.264/H.265 end_of_stream NAL, MPEG-4 Part 2 visual_object_sequence_end_code), the HAL absorbs the marker, drains, and delivers the final onFrameOutput() with endOfStream = true. The client need not additionally set InputBufferMetadata.endOfStream = true, but doing so is not an error - the two EOS sources collapse to a single event.

After the EOS callback the decoder remains in State::STARTED but is drained. No further onFrameOutput() is delivered until flush() or stop() + start().

Decoded Video Frame Buffers

Decoded video frame buffers are only passed from the video decoder to the client when operating in the non-tunnelled operational mode.

If the input AV Buffer that contained the coded video frame was passed in a secure buffer, then the corresponding decoded video frame must be output in a secure video frame buffer.

Video frame buffers are passed back as handles in the IVideoDecoderControllerListener.onFrameOutput() function frameBufferHandle parameter. In tunnelled mode, -1 is passed as the handle value to indicate that no frame buffer handle is being provided since the video is consumed internally by the vendor layer.

The format of the data in the decoded video frame buffer is determined by the vendor driver implementation and does not need to be understood by the RDK middleware.

The frame buffer handle is later passed to the Video Sink for queuing before presentation and is then freed.

The vendor layer is expected to manage the pool of decoded frame buffers privately and report its size in the OUTPUT_FRAME_POOL_SIZE property.

If the frame buffer pool is empty then the video decoder cannot output the next decoded frame until a new frame buffer becomes available. While frame output is blocked, it is reasonable for the video decoder service to either buffer additional coded input buffers or to reject new calls to decodeBufferWithMetadata() with a false return value.

Input Buffer Back-Pressure

IVideoDecoderController.decodeBufferWithMetadata() returns false when the internal decode buffer queue is full. Buffer ownership remains with the caller and the buffer must be retained for re-submission.

To avoid wasted binder transactions, the client SHOULD wait for IVideoDecoderControllerListener.onDecodeBufferAvailable() before calling decodeBufferWithMetadata() again. The callback fires exactly once per back-pressure episode: when the internal queue transitions from full to has-space. If the client continues to call decodeBufferWithMetadata() during back-pressure (receiving false repeatedly), only one callback is delivered per transition. It is not fired in steady-state operation.

Continuing to call decodeBufferWithMetadata() while the queue is full is permitted but will return false repeatedly until space is available.

Presentation Time for Video Frames

The presentation time base units for video frames is nanoseconds and passed in an int64 (long in AIDL definition) variable type. Audio buffers shared the same time base units of nanoseconds.

When coded video frames are passed in through AV Buffer handles to IVideoDecoderController.decodeBufferWithMetadata() the InputBufferMetadata.nsPresentationTime field represents the video frame presentation time.

Calls to IVideoDecoderControllerListener.onFrameOutput() with frame buffer handles (non-tunnelled mode) and/or frame metadata shall use the same nsPresentationTime.

The video decoder shall output frames in presentation order regardless of the order of input frames which is ordered by the encoder.

Subtitling and SEI User Data

To support closed captioning and subtitle extraction, the Video Decoder HAL defines the onUserDataOutput() callback. This callback delivers user data associated with each decoded video frame, typically used to transport caption information embedded in SEI NAL units.

Caption Data Delivery

The user data passed via onUserDataOutput() corresponds to SEI NAL units conforming to the user_data_registered_itu_t_t35 format, commonly used for CEA-608/708 captions:

Begins with ITU-T T.35 country_code (e.g., 0xB5 for USA)
Followed by provider_code (e.g., 0x0031 for ATSC)
Includes a user_identifier string (typically 'GA94')
Contains a user_data_type_code (e.g., 0x03 for CEA-608/708)
Followed by one or more closed caption (CC) data packets

The byte array in onUserDataOutput() starts from the user_identifier field (inclusive).

Timing and Sync

Each onUserDataOutput(nsPresentationTime, userData) call corresponds to a single decoded frame.
The user data must follow the same presentation order as the output video frames to maintain AV sync.
The video frame itself may be delivered before or after the user data.

Client Handling

The RDK media pipeline is responsible for parsing the SEI user data from the byte array to extract CEA-608 or CEA-708 caption packets and synchronize them with playback.

Video Decoder States

The Video Decoder HAL follows the standard Session State Management paradigm.

When an Video Decoder session enters a FLUSHING or STOPPING transitory state it shall free any AV buffers it is holding.

The sequence diagram below shows the behaviour of the callbacks.

sequenceDiagram
    box rgb(30,136,229) RDK Video Decoder
        participant Client as RDK Client
        participant IVideoDecoderEventListener
        participant IVideoDecoderControllerListener
    end
    box rgb(249,168,37) Video Decoder Server
        participant ADC as IVideoDecoder
        participant Controller as IVideoDecoderController
    end
    box rgb(67,160,71) Video AV Buffer
        participant IAVBuffer as IAVBuffer
    end

    Client->>ADC: registerEventListener(IVideoDecoderEventListener)

    Note over ADC: open() transitions from CLOSED -> OPENING -> READY

    Client->>ADC: open(IVideoDecoderControllerListener)
    ADC-->>IVideoDecoderEventListener: onStateChanged(CLOSED -> OPENING)
    ADC->>Controller: new
    ADC-->>IVideoDecoderEventListener: onStateChanged(OPENING -> READY)
    ADC-->>Client: IVideoDecoderController

    Note over ADC: start() transitions from READY -> STARTING -> STARTED

    Client->>Controller: start()
    ADC-->>IVideoDecoderEventListener: onStateChanged(READY -> STARTING)
    ADC-->>IVideoDecoderEventListener: onStateChanged(STARTING -> STARTED)

    Note over Client: Client can now<br>send AV buffers

    Client->>Controller: decodeBufferWithMetadata(bufferHandle=1, {pts, endOfStream=false, ...})
    Client->>Controller: decodeBufferWithMetadata(bufferHandle=2, {pts, endOfStream=false, ...})
    Controller-->>IVideoDecoderControllerListener: onFrameOutput(pts, frameBufferHandle=1000, metadata)
    Controller->>IAVBuffer: free(bufferHandle=1)

    Note over ADC: flush() transitions from STARTED -> FLUSHING -> STARTED

    Client->>Controller: flush()
    ADC-->>IVideoDecoderEventListener: onStateChanged(STARTED -> FLUSHING)
    Controller->>IAVBuffer: free(bufferHandle=2)
    ADC-->>IVideoDecoderEventListener: onStateChanged(FLUSHING -> STARTED)

    Client->>Controller: decodeBufferWithMetadata(bufferHandle=3, {pts, endOfStream=false, ...})

    Note over ADC: stop() transitions from STARTED -> STOPPING -> READY

    Client->>Controller: stop()
    ADC-->>IVideoDecoderEventListener: onStateChanged(STARTED -> STOPPING)
    Controller->>IAVBuffer: free(bufferHandle=3)
    ADC-->>IVideoDecoderEventListener: onStateChanged(STOPPING -> READY)

    Note over ADC: close() transitions from READY -> CLOSING -> CLOSED

    Client->>ADC: close()
    ADC-->>IVideoDecoderEventListener: onStateChanged(READY -> CLOSING)
    ADC->>Controller: delete
    ADC-->>IVideoDecoderEventListener: onStateChanged(CLOSING -> CLOSED)

    Client->>ADC: unregisterEventListener(IVideoDecoderEventListener)