TIPA Camera Tests and the DIN guidelines described in the "Guide for the reliable testing of digital cameras" - 17/01/2013
Recently the German standards organisation (Deutsches Institut für Normung e. V.) announced the availability of a 13-page-paper (DIN SPEC 15707) describing the basic principles for testing digital cameras of any kind (smart phones, compact cameras, DSLR, CSC etc.). Some of these principles may seem self-evident, others might need further discussion (Note: This is not a DIN standard, but a "DIN Spec" =Technical Report), but many of them are quite reasonable.
The test data that TIPA has been offering to its member magazines since 2005, which is produced by BetterNet GmbH, is based on the principles mentioned in the DIN paper, such as creating consistent conditions to make sure that the camera settings chosen for a particular test are those that a normal user would chose with a statistical probability in everyday photography and that those settings allow a comparison between different camera models. All cameras tested for TIPA are evaluated with test photos taken with the standard settings of the camera. Of course, a constant and reproducible lighting situation with daylight characteristics (ca. 5.500 K) is used for testing. Other requirements of the DIN paper have also been met since the beginning of the testing activities, e.g. the use of calibrated and profiled high-end monitors.
The DIN guideline recommends using measurement methods and test charts at least one category "better" than the best result to be expected in testing. This is covered, for example, by using the 41 step "Stouffer Chart" that provides a dynamic range up to 13,3 for dynamic range measurement - far more than actual cameras can deliver today.
Even if the visual (and therefore, subjective) evaluation of test images by a highly experienced tester is considered to be very important by BetterNet, taking into account that photography and image quality are sensual entities that cannot be fitted entirely in lab-measurable criteria, there is also - as the DIN guidelines strongly advise - a software-based (and therefore, objective) analysis for every camera test, measuring criteria such as resolution, noise, chromatic aberration, white balance, vignetting/light falloff, lens distortion, colour reproduction, over- and under-sharpening etc. The software suite employed for the measurements is Imatest, considered to be the "Imaging Industry Standard for Research & Development", and used by companies such as Apple, Samsung, Adobe, Canon Europe, Foveon, Matsushita (Panasonic) and many more (www.imatest.com).
Additional information: http://bit.ly/UFIM6G.