JHOVE white balance error

[stephaniechristensen]

Dev Effort

1D

Description

We are running JHOVE (Rel. 1.20.1, 2018-03-29).
JHOVE defines these values:
0=Auto white balance
1= Manual white balance
The white balance mode is set when the image is shot. It appears newer camera software, the white balance settings are out of the range of JHOVE acceptable values. We've seen values such as 4,5, 7 thus throwing an error. Auto=0 Manual=1. As the camera software distinguish between Auto, Daylight, Tungsten, Flash, etc. different numbers are assigned.
The files that are failing due to the white balance error are "well-formed" but not "valid". We've had other errors in the "is valid" that we want to alert us to, so if we skip it, we skip checking if the TIF is well-formed and valid. We can look to parse out that parameter, but will require a lot more intervention/overhead on our end.
Other imaging software render the files fine. JHOVE appears to be too rigid in failing the white balance values giving newer camera software variables that result in different balances.
New cameras seem to increasingly be sending us values outside the accepted range, and we are picking up these types of errors from images that are captured from multiple sources.
My understanding is earlier versions of JHOVE, "white balance" was not checked. It's with JHOVE 1.18 and on this is now checked. Does JHOVE plan to update the values for more comprehensive white balance reporting? Does JHOVE plan to support the expanded white balance values in JHOVE1?

[tledoux]

In fact, prior to 1.18, the "white balance" was not extracted and just ignored. Now that i't's extracted it tries to map with the defined values by the standard JEITA CP-3451D 2.3 which defines only the 2 values (0= Auto white balance, 1 = Manual white balance), any other value is considered an error.
But in fact reading more carefully the standard, it states "Other = reserved" ; so indeed other values should be accepted but can't be decoded. So the way these fields are checked should probably be amended.

It could be very interesting if you could provide a sample containing other values and if you have a reference on such usage (list of values used by some cameras).
Thanks

[tledoux]

My current understanding is that the additional information on the kind of light should go to the EXIF tag LightSource (37384 or 0x9208) where the values of DayLight, Tungsten, ... are defined.
But maybe this comes from a post-producion software ?

In any case, in order to take those values (which are out of the current standards), we definitively need some real samples in order to correctly add a parameter to allow for out of spec values and implement it in the right way.

[stasiaknyu]

Hello! Was very relieved to come across this issue. My lab both creates and receives files produced on PhaseOne cameras that "failed" Jhove 1.20 in the white balance field. It seems to me to be slightly arbitrary to decide that some forms of white balance are acceptable but others are not, but I have to admit that my Jhove knowledge is slim, so any help or corrections are appreciated.

We rolled back to Jhove 1.18, but are wondering if newer versions of Jhove will correct to accept other white balance field values? Thanks.

[jgpawletko]

Hello All,
I work with @stasiaknyu at NYU.
Per @tledoux's request, attached is a file with the following White Balance property:

       <property>
        <name>WhiteBalance</name>
        <values arity="Scalar" type="Integer">
         <value>5</value>
        </values>
       </property>

Please let me know if you need additional info.
Thank you!

princeton_aco002370_000010.tif.gz

(please note: I had to gzip the TIFF file in order to upload it.)
MD5 checksum of TIFF:
e5db4a5670cbb4985ac8d022a0c0e96b princeton_aco002370_000010.tif

[tledoux]

As stated before the latest standard on Exif 2.31, only accepts 2 values for WhiteBalance (0=Auto white balance,1 = Manual white balance); the other values are proprietary and can't be decoded...
So for Jhove in the current state of its knowledge, the image couldn't be "Valid" (conforming to a known standard).

However, you're correct saying that the image should be parsed without error (the bug in Jhove should be corrected by PR #449) : the image should be "Well-formed" and an error/warning message should inform that the value 5 is out of range in the "WhiteBalance" property.

[leninoc]

Hi all, we at Archives New Zealand run into same issue, whitebalance value we have in externally digitised tifs is 5. Was not problem before (Jhove 1.17) as explained in this thread, but its a problem now resulting in Well formed but not valid in JHOVE 1.20 (tif hul [1.8).] Strangely when using JHOVE 1.22 (tif hul 1.9.1) the results are different. First of all JHOVE wont automatically use tif module, but uses Bytestream. When manually set to use tif module, the message is "ErrorMessage: Validation ended prematurely due to an unhandled exception." The image (one of many we have) is here https://1drv.ms/u/s!Akbfi-DUwSXdid9aYfiuB-76STVV5g?e=aqdxAN
edit- it actually looks like what is known as issue #431

[MartinSpeller]

JHOVE white balance error #364 - Assigned to TBA

[thorsted]

I can confirm the same issue with images from a PhaseOne/CaptureOne system. It seems to me like a bug on their end, the additional values should be in Light Source not White Balance. That said, I am curious if changing this to a InfoMessage would be better?

[thepowerthatbe]

Please see the below technical bulletin from my company Digital Transitions (DT Heritage) from 2018.

---

Issue: JHOVE 1.18 will fail TIFFs from some cameras, including Phase One cameras, if the white balance is set to anything other than Auto or Daylight.

Explanation: The acceptable standards-based values accepted for the WhiteBalance tag in JHOVE version 1.18 and later include: Auto=0 and Manual=1. If the camera was set to "Auto White Balance" upon capture, the downstream TIFF would register a value of "Auto" or "0" for this field and pass. Likewise when set to "Daylight White Balance" upon capture (As Shot), the downstream TIFF would register a value of "Manual" or "1" for this field and pass. However, when a camera is set to any other white balance setting the downstream TIFF would contain a value other than 0 or 1, and be flagged as failing.

Some camera manufacturers differentiate between Auto, Daylight, Flash, Tungsten, Fluorescent, Custom or other named white balances upon capture (As Shot). For instance, in some cameras "Flash" might receive an assigned a value of "4" and "Custom" a value of "6." Since the standard only allows for Auto and Daylight this is technically not allowed by the standard.

Earlier versions of JHOVE (through version 1.16), "WhiteBalance" as a tag was not strictly checked by JHOVE. This meant JHOVE would pass WhiteBalance values other than 0 or 1. In JHOVE version 1.18 and later, the "WhiteBalance" tag was more strictly checked. Consequently, any file that captured with the "As Shot" white balance set to anything other than "Auto" or "Daylight," the file will automatically register as not valid because JHOVE is restricting this tag to a value of only "0" or "1."

We see four ways users can respond to this JHOVE flag:

1. Lobby for the specification to be updated
   Ideally the international specification would be updated to acknowledge the current industry practice of using values higher than 1 to retain knowledge of a specific non-automatic white balance setting, and JHOVE would then be updated to acknowledge values allowed by the updated specification.

2. Use only camera settings that comply with the standard (and therefore JHOVE)
   You can keep the White Balance on your camera set in such a way that white balance tag in the file is set to 0 or 1. For example on a Phase One iXG you can set the camera to "Auto" white balance in the camera menu through the back of the LCD screen. If you're using Capture One you can use the white balance dropper to set the white balance on the first captured image in a session, and use the "Copy from last" feature in the "Next Capture Adjustments" tool tab to carry this white balance throughout all subsequent captures.

3. Modify the TIFFs after they are made
   We have created a droplet script (attached) that will force the WhiteBalance tag to "Manual" and will bring the already processed TIFF files inline with JHOVE standards (attached). To use it, drag and drop the files onto the droplet to make the change. The script is simple, open source, and only requires EXIF Tools to be installed. Anyone is welcome to use it as a starting point for their own custom version. Just email me for a copy (posting executables might flag spam filters).

4. Set JHOVE to ignore this flag
   It is difficult to imagine a situation in which this value would cause any real-world issues. JHOVE exists to check the strict adherence of a given file to that file's specification as-written, but it is up to the user to decide if any such deviations represent a threat to their workflow and preservation needs. If the user decides this tag does not represent such a threat, they can simply tell JHOVE not to flag it, as was the case in JHOVE version 1.16 and earlier.

Updating JHOVE to flag (info) rather than fail is also an option. But my feeling is that goes a bit against JHOVE's core purpose. Such files does not comport with the standard, however pedantic the non-compliance may be. But I'm not an expert in JHOVE.

[thorsted]

@thepowerthatbe Has DT along with CaptureOne considered using the tags correctly and using the Light Source tag to record the actual light source? Leaving White Balance to be Auto or custom?
TIFF Tag LightSource

[thepowerthatbe]

The underlying issue is the firmware of the camera rather than software such as Capture One. 

The camera firmware is writing a value that does not strictly comply with the standard. Capture One software is just dutifully passing on that value. This is true of Phase One cameras and also true of several other brands.

We have previously suggested to Phase One they change the way this tag is written, and will remind them of that suggestion now. I do not have suitable deep technical contacts at other camera makers, but if anyone here does I would encourage them to do the same.

[mel-mason]

We also have this issue. For us, setting JHOVE to ignore this flag would probably be the best option - I'm hesitant to bulk EXIF-update the amount of TIFFs we have, as they are our preservation copies. Does JHOVE support ignoring specific flags? I couldn't find documentation on how to configure that.

[carlwilson]

Does JHOVE support ignoring specific flags? I couldn't find documentation on how to configure that.

Not as things stand. The upcoming 1.28 release marked the unification of error reporting to an improved framework that supports this. The intention is to implement error filtering in the next release where users could supply a list of error IDs to be suppressed, and/or perhaps given another status.

[thorsted]

I was just reading through the newly released EXIF 3.0 specification and nothing has changed in regards to White Balance, there should only be Auto or Manual. I hope the ability to suppress this error is still on the table.
https://www.iptc.org/news/exif-3-0-released-featuring-utf-8-support/

[DasFaultier]

Hi everyone,

we just ran into this problem for the first time with JHOVE 1.28.0, TIFF-hul 1.9.4.

We found multiple TIFF images that contained non-Exif-compliant metadata and failed in Jhove. The tiffinfo output of the ExifIFD shows:
WhiteBalance: 5

We also see hints of PhaseOne cameras and software being used to create the TIFFs:

Make: Phase One
Model: IQ4 100MP Trichromatic
Software: Capture One 22 Windows

Similar information can be found in the XMP metadata.

We have discussed the issue internally and came to the conclusion that both PhaseOne and Jhove don't behave correctly in respect to the Exif specifications:

• PhaseOne writes values that are in the reserved range for WhiteBalance and cannot be mapped to the available string values.
• Jhove fails to gracefully handle this condition as is should. Reserved values are reserved for further use in future versions of the Exif spec and should be ignored by parsers as long as they're not properly specified, even if they're in unofficial use with some manufacturers.

I'm not a Java dev myself, but upon inspection of the available code, I would assume that the relevant portion can be found at

jhove/jhove-modules/tiff-hul/src/main/java/edu/harvard/hul/ois/jhove/module/tiff/ExifIFD.java

Line 516 in 9880643

if (_whiteBalance != NULL) {

(and below).

The current code reads:

        if (_whiteBalance != NULL) {
            entries.add (addIntegerProperty ("WhiteBalance", _whiteBalance,
                                             WHITEBALANCE_L, rawOutput));
        }

But it should be something like:

        if (_whiteBalance != NULL && 0 <= _whiteBalance <= 1) {
            entries.add (addIntegerProperty ("WhiteBalance", _whiteBalance,
                                             WHITEBALANCE_L, rawOutput));
        }

If you agree with this, would someone with more Java skills and repo merge permissions please edit the code accordingly?

Hint: A similar repair might be necessary for some/all other places in the same file where addIntegerProperty() is used. I haven't checked all of them, but at a glance I noticed the SensingMethod property at

jhove/jhove-modules/tiff-hul/src/main/java/edu/harvard/hul/ois/jhove/module/tiff/ExifIFD.java

Line 493 in 9880643

entries.add (addIntegerProperty ("SensingMethod", _sensingMethod,

which may only contain values 1 <= _sensingMethod <= 8, but is of type SHORT. SHORT is specified in Exif as "A 16-bit (2-byte) unsigned integer", and the variable is implemented as proper Java int, so 32-bit unsigned. Given that Java doesn't have unsigned integer types und the available space is far larger that what is specified in Exif, it's probably sensible to use smaller integer types and check for specified ranges.

Thank you and kind regards,
Jörg

EDIT: Adding the piece of Exif spec that says PC software should ignore things that are not specified in Exif. From Exif 3.0, section "3.5. Presupposed Systems and Compatibility" (bold font emphasis added by me):

"A basic condition of equipment with a playback function is that it shall read the files it recorded itself. In a personal computer environment, the basic requirement is that dedicated software shall read all the images, audio and attribute information recorded in conformance with this standard"

That means that any information that is not in conformance with the standard shall not be read.

EDIT 2: I just realized after the fact that Java's short data type doesn't match Exif's SHORT, because one is unsigned and the other one isn't, so in order to cover the whole SHORT space, it's necessary to at least use Java int. Please disregard that part above.