L4 Smoke Test (VST)

Overview (VST)

This document outlines the automated test plan for L4 (Vendor Full Stack) testing of RDK Video releases. These tests focus on verifying the stability and functionality of the RDK Video stack by testing the integration of the vendor, application (APP), and middleware (MW) layers.

Goal: To automate the execution of smoke and regression tests that are currently performed manually, improving efficiency and reducing testing time.

Scope: These tests are designed as "big ticket" checks, focusing on core functionality:

• Is video running?
• Is audio running?
• Is video/audio sync operating correctly?
• Are trick modes operational? (Including checks for black screens, display movement, and errors)

These tests are not intended to be deep-dive device analysis tests. Those are covered by other testing activities. The focus here is on smoke testing to validate that main functionality is still operational after:

• Integrating a new vendor layer with the existing APP/MW.
• Integrating a new APP/MW layer with the existing vendor layer.

This ensures that any new layer introduced does not break the core functionality of the stack.

Framework: python_raft will be used to automate these tests, enabling engineers and Jenkins to trigger them as needed.

Test Approach:

• Build: A full stack image is created using the relevant combination of  APP/MW and Vendor layers.
• Automation: python_raft scripts will be developed to automate the test cases described below.
• Execution: Tests can be triggered by engineers or integrated into the Jenkins CI/CD pipeline.
• Reporting: python_raft will provide test results and logs for analysis.

Important: These tests are designed as a pre-release quality gate, ensuring no regressions are introduced before a new layer (vendor, APP, or MW) is released.

Target Environment Assumptions:

• Full Stack Image: The testing suite can assume that the target device has been programmed with a valid full stack image (vendor, middleware, and application layers).
• System Ready: Tests should wait for the system to be fully operational before commencing. python_raft already provides mechanisms to achieve this (e.g., by monitoring system logs or specific services, see waitForBoot() ).

Configuration and Platform Independence:

• Configuration-driven: The testing suites will be driven by configuration information on the CPE and the image that's running. This includes details about the platform, software version, and available features. To be clear python_raft already supports this, and has methods to extend based on requirements.
• Platform Abstraction: Tests should prioritize launching applications from the command line whenever possible. This approach offers several advantages:
  • Consistency: Command-line interfaces tend to be more stable across platforms and software versions compared to graphical interfaces and remote control key sequences.
  • Speed: Launching applications from the command line is generally faster than navigating through menus using a remote control.
  • Reliability: Command-line launching eliminates potential issues with IR/Bluetooth signal interference or key sequence misinterpretations.
• Key Playback Classes (for specific scenarios): While command-line launching is preferred, there might be scenarios where using IR/Bluetooth keys is unavoidable. In such cases:
  • Wrap platform-specific key sequences in key playback classes. These classes will abstract the underlying key sequences, providing a consistent interface for test scripts.
  • Drive the testing suites by versioned profiles based on the platform. These profiles will contain the necessary configuration data for each platform and software version, ensuring that the correct key sequences and navigation paths are used.

By prioritizing command-line application launching and utilizing key playback classes and versioned profiles when necessary, the testing suite can achieve a high degree of platform independence and resilience to changes in the user interface and remote control configurations.

Requirements:

1. Integration with L4-wide Validation Classes: The L4 testing suite shall integrate with common L4-wide classes that perform validation checks. These classes will abstract the underlying validation mechanisms, allowing for a phased approach to automation:
2. Phase 1: Human-assisted Validation: Initially, these classes may prompt a human tester with a simple yes/no question to confirm the expected behaviour.

3. Phase 2: Automated Validation:  These classes will be progressively upgraded to perform automated validation by directly interacting with the system, e.g., checking proc files for decoder activity, analysing log output, or querying system states. This allows for a gradual transition from manual to automated testing.

4. Engineering-focused Debugging: This test suite is primarily intended for use by engineers. When test failures occur, the framework should facilitate debugging by:

5. Single-stepping through Python Orchestration:  Engineers should be able to easily single-step through the python_raft scripts to understand the test flow and pinpoint the failing steps.
6. Seamless Access to Target Device: The framework should provide mechanisms for engineers to readily access the target device (e.g., via SSH) for debugging purposes. This allows them to analyse C code, examine driver data, and inspect system logs in parallel with the test execution.

By adhering to these requirements, the L4 automated testing suite will not only provide a robust quality gate but also empower engineers to efficiently identify and resolve issues across the entire stack.

Test Suites

A. Linear Channels & VOD

Test Case	Description	Status (Pass/Fail)	Remarks
A1	General Sanity on APP Playchannels: Play channels for 5 minutes, changing channels via remote.
A2	Sanity on Linear SDR Channel: Test channel 101/501.
A3	Sanity on Linear UHD Channel: Test channel 406.
A4	Sanity on Linear Channel Requiring PIN: Test channels 301, 302.
A5	Sanity on VOD (SDR): Test SDR VOD content.
A6	Sanity on VOD (UHD): Test UHD VOD content.

B. Apps Testing

Test Case	Description	Status (Pass/Fail)	Remarks
B1	APPS - YouTube - SDR: Test YouTube with SDR content.
B2	APPS - YouTube - HLG: Test YouTube with HLG content.
B3	APPS - YouTube - HDR10: Test YouTube with HDR10 content.
B4	APPS - Netflix - SDR: Test Netflix with SDR content.
B5	APPS - Netflix - DV: Test Netflix with Dolby Vision content.
B6	APPS - Disney+ - SDR: Test Disney+ with SDR content.
B7	APPS - Disney+ - DV: Test Disney+ with Dolby Vision content.
B8	APPS - Amazon Prime - SDR: Test Amazon Prime with SDR content.
B9	APPS - Amazon Prime - HDR10: Test Amazon Prime with HDR10 content.
B10	APPS - Amazon Prime - DV: Test Amazon Prime with Dolby Vision content.
B11	APPS - Apple TV - DV: Test Apple TV with Dolby Vision content.
B12	APPS - Paramount+ - SDR: Test Paramount+ with SDR content.
B13	APPS - YouTube Kids - SDR: Test YouTube Kids with SDR content.
B14	APPS - BBC iPlayer - SDR: Test BBC iPlayer with SDR content.
B15	APP - Peacock: Test Peacock app.

C. Switching Between Apps

Test Case	Description	Status	Remarks
C1	YouTube to: Netflix, Amazon Prime, Disney+, Apple TV, Paramount+, YouTube Kids, Peacock.
C2	Netflix to: YouTube, Amazon Prime, Disney+, Apple TV, Paramount+, YouTube Kids, Peacock.
C3	Disney+ to: YouTube, Amazon Prime, Netflix, Apple TV, Paramount+, YouTube Kids, Peacock.
C4	Apple TV to: YouTube, Amazon Prime, Netflix, Disney+, Paramount+, YouTube Kids, Peacock.
C5	Paramount+ to: YouTube, Amazon Prime, Netflix, Disney+, Apple TV, YouTube Kids, Peacock.
C6	YouTube Kids to: YouTube, Amazon Prime, Netflix, Disney+, Apple TV, Paramount+, Peacock.
C7	Peacock to: YouTube, Amazon Prime, Netflix, Disney+, Apple TV, Paramount+, YouTube Kids.
C8	Amazon Prime to: YouTube, Netflix, Disney+, Apple TV, Paramount+, YouTube Kids, Peacock.

D. Wi-Fi / Ethernet Test

Test Case	Description	Status(Pass/Fail)	Remarks
D1	Factory Reset and Connect: Factory reset the panel and connect to 5GHz/2.4GHz SSID during setup.
D2	Connect Ethernet: Connect Ethernet while the panel is on 5GHz/2.4GHz Wi-Fi.
D3	Disconnect Ethernet: Disconnect Ethernet and check if it resumes the last Wi-Fi connection.
D4	Verify IP Address: Check if the panel's IP address matches the one assigned by the router.
D5	Play/Stream Videos: Play/stream 4K/HDR/UHD videos on 5GHz/2.4GHz network.
D6	Soft Reboot (Wi-Fi): Soft reboot the panel while connected to 5GHz/2.4GHz and ensure reconnection.
D7	Boot Up Time (Wi-Fi): Check boot-up time when connected to 5GHz/2.4GHz.
D8	Hard Reboot (Wi-Fi): Hard reboot the panel while connected to 5GHz/2.4GHz and ensure reconnection.
D9	Boot Up Time (Wi-Fi): Check boot-up time when connected to 5GHz/2.4GHz.
D10	Reset Network: Reset the network while the panel is on 5GHz/2.4GHz.
D11	Soft Reboot (No Internet): Soft reboot after network reset and ensure the panel boots with no internet.
D12	Boot Up Time (No Internet): Check boot-up time with no internet.
D13	Hard Reboot (No Internet): Hard reboot after network reset and ensure the panel boots with no internet.
D14	Boot Up Time (No Internet): Check boot-up time with no internet.
D15	Disable Wi-Fi: Disable Wi-Fi via settings and ensure the "no internet" screen appears.
D16	Hard Reboot (No Internet): Hard reboot and ensure the panel boots with no internet.
D17	Disable Wi-Fi: Disable Wi-Fi via settings and ensure the "no internet" screen appears.
D18	Soft Reboot (No Internet): Soft reboot and ensure the panel boots with no internet.
D19	Enable Wi-Fi: Enable Wi-Fi and ensure the panel connects to the last Wi-Fi connection.
D20	Power Cycle Router: While connected to 5GHz/2.4GHz, power cycle the router and ensure the panel reconnects.
...	...	...	...

E. Motion & Deep Sleep Test

Test Case	Description	Status (Pass/Fail)	Remarks
E1	Verify Motion Events: Verify motion events are detected.
E2	HDMI Content and Motion: Play HDMI content and verify motion events.
E3	Linear Channel and Motion: Play linear channels and verify motion events.
E4	Apps and Motion: Play apps like YouTube/Netflix/Disney+ and verify motion events.
E5	Power Cycle and Motion: Power cycle the panel and check motion events.
E6	Standby and Motion: Put the panel in standby, wake it up, and check motion events.
E7	Deepsleep and Motion: Put the panel in deep sleep, wake it up, and check motion events.
E8	Auto-Standby: Ensure the panel enters standby mode after 50 minutes of inactivity.
E9	Banner Removal: Check if the banner is removed by generating
motion at the 51st minute.
E10	Linear Channel and No Banner: Ensure no banner appears after 50 minutes of activity on linear channels.
...	...	...	...

F. LED Behaviour Test

Test Case	Description	Status (Pass/Fail)	Remarks
F1	Checking LED behaviour on power cycle from ON state
F2	Checking LED behaviour on power cycle from standby mode state
F3	Checking LED behaviour on power cycle from Deepsleep mode state
F4	Checking LED behaviour on Soft reboot the panel from ON state
F5	Checking LED behaviour on Soft reboot the panel from standby state
...	...	...	...

G. Remote Test

Test Case	Description	Status (Pass/Fail)	Remarks
G1	Wake panel via IR remote --HOME button
G2	Wake panel via IR remote --Power button
G3	Wake panel via IR remote --PRIME Button
G4	Wake panel via IR remote -NETFLIX button
G5	Navigate the EPG page, try trick modes & different buttons on remote (IR)
G6	Pair the remote on BT
G7	Unpair the remote & pair it again
G8	Navigate the EPG page, try trick modes & different buttons on remote (BT)
G9	Wake panel from standby mode via BT remote --HOME button
G10	Wake panel from standby mode via BT remote --Power button
G11	Wake panel from deep sleep mode via BT remote --HOME button
G12	Wake panel from deep sleep mode via BT remote --Power button

H. FFV Test

Test Case	Description	Status (Pass/Fail)	Remarks
H1	Mic enable via side button - Launch channel, apps, volume etc via FFV
H2	Mic enable via side button - Put panel in standby via FFV
H3	Mic enable via side button - Wake panel via FFV from standby
H4	Mic enable via side button - Put panel in deep sleep & wake it via FFV
H5	Mic enable via side button - Soft reboot panel & check if MIC status remains persistent
H6	Mic enable via side button - Hard reboot panel & check if MIC status remains persistent
H7	Mic disable via side button - Launch channel, apps, volume etc via FFV
H8	Mic disable via side button - Put panel in standby via FFV
H9	Mic disable via side button - Wake panel via FFV from standby
H10	Mic disable via side button - Put panel in deep sleep & wake it via FFV
H11	Mic disable via side button - Soft reboot panel & check if MIC status remains persistent
H12	Mic disable via side button - Hard reboot panel & check if MIC status remains persistent
...	...	...	...

I. HDMI Source - FireTV / Chromecast / AppleTV

Test Case	Description	Status (Pass/Fail)	Remarks
I1	Launch HDMI source via HDMI Source remote
I2	Play any content on HDMI source & check AV
I3	Change different resolution on HDMI source
I4	When panel is in standby, wake panel via HDMI source
I5	When panel in deep sleep, wake panel via HDMI source
I6	Hot plug HDMI source
I7	Switch between HDMI source, APPS

J. HDMI Audio Device

Test Case	Description	Status (Pass/Fail)	Remarks
J1	Connect HDMI speaker on HDMI & check sound coming from speakers
J2	Hot plug HDMI speakers & make sure Audio switches back properly
J3	Check any AV sync issue
J4	Check audio for contents on HDMI source when HDMI speakers connected
J5	Put panel in standby & wake it up, check the audio coming from HDMI speakers
J6	Put panel in deep sleep & wake it up, check the audio coming from HDMI speakers

K. BT Audio Device

Test Case	Description	Status (Pass/Fail)	Remarks
K1	Pair BT speakers & check sound coming from speakers
K2	Check any AV sync issue on BT speakers while playing apps, HDMI, linear channel
K3	Check audio for contents on HDMI source when BT speakers connected
K4	Change the output of audio via quick menu
K5	Put panel in standby & wake it up, check the audio coming from BT speakers
K6	Put panel in deep sleep & wake it up, check the audio coming from BT speakers

L. DTT Channels

Test Case	Description	Status (Pass/Fail)	Remarks
L1	Checking AV of DTT channel (SD & HD channels)
L2	Scanning DTT channels
L3	Switch between DTT channels & HDMI source
L4	Browsing TV guide, signal information, audio etc in DTT channels option

M. Boot Up Time

Test Case	Description	Status (Pass/Fail)	Remarks
M1	Hard reboot (power cycle) 1
M2	Hard reboot (power cycle) 2
M3	Hard reboot (power cycle) 3
M4	Soft reboot (/rebootNow.sh) 1
M5	Soft reboot (/rebootNow.sh) 2
M6	Soft reboot (/rebootNow.sh) 3