Conviva Precision for A/B testing
Why use A/B testing
Let’s say you are considering introducing a higher bitrate stream for increased picture fidelity. Given the extra costs in Encoding, Origin and CDN, this is a decision that needs a clear and justified business driver. Our customers usually ask:
1. What's the industry trend on average bitrates for similar types of service, content and devices? How does this trend evolve over time and what is my competition doing?
2. How can I calculate the ROI and justify the costs? Will this decision result in higher viewer engagement and consequently audience?
For a long time we have been able to help with #1 using Experience Benchmarks, which enables smarter decision making. It gives publishers unmatched insight and intelligence into streaming experience performance against their peers.
The best way to help with #2 is with AB group testing, which allows you to compare the results in terms of QoE and Engagement on each group.
Why use Precision
For A/B viewer testing, you create two groups:
- group A are viewers for a particular series or live channel, always getting a higher bitrate stream
- group B are other viewers always getting streams with assigned lower bitrate.
Precision will allow you to allocate the streams to groups A and B consistently. It also automatically ties with Experience Insights so you can confirm the QoE KPIs on each group of viewers and respective Viewer Engagement. Precision can assure that you have a meaningful engagement test, where the groups A and B MUST remain consistent (a viewer cannot get the lower bitrate stream sometimes and the higher another times). Then, Experience Insights will let you evaluate the results in terms of QoE and Engagement, for each group. Advanced analytics reporting capabilities can be leveraged to determine the statistical significance of the results.
How does it work
To achieve this viewer demarcation, Precision uses the same technology that already allows Publishers to Optimize by what we call a Generic Resource.
A Generic Resource is typically a CDN, or a CDN using a given Origin, or a given PoP from a CDN. In an example scenario of AB testing, the Generic Resource would be "Live Channel X with a Maximum Bitrate of 4Mbps on Roku" and "the same Channel but with 6M on Roku". In this case there is no concept of Optimization, but only load balancing two streams consistently and being able to connect this allocation to Experience Insights.
For further discussions on how to explore Precision capabilities, please contact us.
Enhancements to MetricLens Drill-Downs
and TimeLine Analysis
Conviva’s MetricLens dashboards enable you to perform advanced analysis of QoE and audience data through a combination of filters, data ranges, metric values, and dimensions. This combination of metrics and performance data allows you to quickly understand the most critical contributors - ISPs, cities, player versions, assets – of your video streaming QoE and engagement issues.
To transition from summary data to time series analysis, MetricLens users are able to easily navigate between MetricLens summary and Timeline dashboards to quickly analyze timelines of the top five dimension values, determined by the number of Attempts. With the latest Pulse release, Pulse users can now also quickly analyze a timeline of each individual dimension to discover the detailed timing of anomalies in any dimension value.
Top Five Dimension Values
From a MetricLens Summary page, use the View Timeline button to view the time series of the top dimension values. MetricLens displays the metric-based time series with filters for each of the top five dimension values, in this example the top DMAs show metric spikes on Thursday 04:00.
The View Summary button returns to the MetricLens Summary page.
Single Dimension Value
To drill-down to the time series for a single dimension value for more specific dimension analysis, simply click the linked dimension value on a summary page. MetricLens displays the metric-based time series filtered for that dimension value, in this example focusing the time series analysis on the QoE metrics for the most impacted DMA, New York.
These MetricLens navigation paths are supported in both Conviva Experience Insights and Ad Insights MetricLens pages.