Company:VCX score

Short description: Benchmark for camera testing

VCX Score

VCX logo
Trade name	VCX
Area served	Global
Website	vcx-forum.org

VCX score is a smartphone camera benchmarking score described as "designed to reflect the user experience regarding the image quality and the performance of a camera in a mobile device". developed by a non-profit organization - VCX-Forum.

VCX scores are used by specialist media and by VCX-Forum members to showcase the benchmarking of smartphones, as well as market photography technology.

VCX scoring methodology has been cited in various published books and independent imaging organizations:

• Book: Camera image quality benchmarking ISBN:978-1-119-05449-8

• Article in Journal of Electronic Imaging - VCX: An industry initiative to create an objective camera module evaluation for mobile devices.
• Article in Journal of Electronic Imaging - VCX Version 2020: Further development of a transparent and objective evaluation scheme for mobile phone cameras.

Service Provider

VCX-Forum (where VCX is an acronym for Valued Camera eXperience) is an independent, non-governmental, standard-setting organization for image quality measurement and benchmarking (VCX score). Its members are drawn from mobile phone manufacturers, mobile operators, imaging labs, mobile and computer chipset manufacturers, sensor manufacturers, device manufacturers, software companies, equipment providers, and camera & accessory manufacturers among others.

VCX-Score methodology

Tenets

VCX score methodologies are based on the 5 Tenets:

• VCX-Forum test measurements shall ensure the out-of-the-box experience
• VCX-Forum shall remain 100% objective
• VCX-Forum shall remain open and transparent
• VCX-Forum shall employ an independent imaging lab for testing
• VCX-Forum shall seek continuous improvement

Parameters

To ensure the test results accurately reflect the user experience, the image quality is evaluated for five parameters:

• Spatial Resolution
• Texture loss – the ability of the device to reproduce low contrast, fine details
• Sharpening – the ability of the device to sharpen with minimum distracting artifacts
• Noise – the ability of the device to suppress noise while minimizing obfuscation of details
• Dynamic range – the ability of the device to capture maximum contrast in a scene
• Color Reproduction – the ability of the device to capture colors closely matching the original scene

Setup

• The device under test is mounted on a tripod on rails to keep the reproduction scale constant between devices under test
• The entire lab is temperature-controlled to standard room temperature (23 °C ± 2 °C)
• The device under test is expected to:
  1. reproduce reflective test targets like the "TE42-LL" (TE42-LL target in A1066 and A 460 (Selfie) in 4:3 and 16:9);
  2. reproduce transmissive TE269B test target (for dynamic range measurements); and
  3. reproduce test charts while mounted on a hand simulation device (a device which simulates the movement of the human hand to measure the motion stabilization apparatus of the device, based on ISO 20954-2).
• The device under test is then used to capture a series of images and video in various controlled lighting conditions

A detailed description of the setup and procedure is available as a whitepaper on the VCX-Forum website. as well as in the book, Camera Image Quality Benchmarking, page 318, section 9.4.3

Labs and testing

Tests and benchmarks are conducted by independent labs. The test procedure, metrics, and weighting are dictated by the standard developed by VCX-Forum.

Benchmark publication

VCX scores are published on the VCX-Forum website. Parts of this publication are often reproduced in specialist media and smartphone vendor social media channels as part of their marketing campaign.

Criticism

Metrics and weighting

VCX-Forum claims that all test measurements must ensure the out-of-the-box experience (Tenet 1 of VCX-Forum) but does not specify what happens when the devices are updated later on.

VCX-Forum claims to be objective (Tenet 2 of VCX-Forum) but uses subjective components for the formation of the weighting itself. This subjective base is claimed to have come from blind tests for which no evidence has been provided on the website.

Despite the claim that VCX is an open and transparent standard (Tenet 3 of VCX-forum), the details of weighting and scoring are only visible to members of the VCX-Forum.

Most measurements are done with the device on a tripod and aimed at test charts. This does not reflect the common user scenario that VCX-Forum claims to reflect.