Author Topic: More detailed JPEG Compression info  (Read 934 times)

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 745
  • Professional Photographer and Workshop Leader
    • Hayo Baan – Photography
More detailed JPEG Compression info
« on: January 09, 2017, 03:31:05 AM »
I was really surprised (nay, shocked: I don't want my raw data to be lossy) by the fact that the raw data inside a DNG was reported as JPEG compressed. Since I changed my workflow to DNG, it was very important to me to be sure I wasn't losing any raw data information. So I started to read up on the DNG standard a bit more and found something really interesting: JPEG can also be lossless!

In a DNG, depending on the Photometric Interpretation and Bits Per Sample, the specification says the JPEG compression must either be “baseline DCT JPEG” (lossy) or “lossless Huffman JPEG”.

It would be nice if exiftool told us this without having to derive this ourselves. This would require an enhancement to the extraction of the Compression tag(s), also incorporating the values of the other tags in the final result.

I can, of course, also create a custom (composite) tag to do this, but the problem is that there are multiple SubIFDs that all can have a different Compression/PhotometricInterpretation/BitsPerSample, so this doesn't work well.

Hayo Baan – Photography
Web: www.hayobaan.nl
Blog: blog.hayobaan.nl
500px: 500px.com/HayoBaan

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 11374
    • ExifTool Home Page
Re: More detailed JPEG Compression info
« Reply #1 on: January 09, 2017, 07:24:45 AM »
Hi Hayo,

It would be nice if exiftool told us this without having to derive this ourselves. This would require an enhancement to the extraction of the Compression tag(s), also incorporating the values of the other tags in the final result.

What other tags?  This does sound like a job for a Composite tag, but it may require a bit of advanced wizardry.

- Phil
...where DIR is the name of a directory containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 745
  • Professional Photographer and Workshop Leader
    • Hayo Baan – Photography
Re: More detailed JPEG Compression info
« Reply #2 on: January 09, 2017, 07:58:42 AM »
What other tags?  This does sound like a job for a Composite tag, but it may require a bit of advanced wizardry.

I meant Photometric Interpretation and Bits Per Sample: If PhotometricInterpretation = 6 (YCbCr) and BitsPerSample = 8/8/8, or if PhotometricInterpretation = 1 (BlackIsZero) and BitsPerSample = 8, then it's lossy JPEG. In all other cases is's lossless JPG. It would be nice if this could be noted alongside the Compression tag. (See page 19 of the DNG spec on Compression)

The problem with a composite tag is that you can only define one, but as I said there are multiple SubIFDs that have the Compression tag so this doesn't really work :(

Note: I tried to create my own $Image::ExifTool::Exif::Main{0x103}->{PrintConv} but as soon as I call GetInfo there to attempt to get e.g. the matching PhotometricInterpretation ExifTool no longer shows tag values…
Trial code in .ExifTool_config:
Code: [Select]
require Image::ExifTool::Exif;
$Image::ExifTool::Exif::Main{0x103}->{PrintConv} = q{
    my $grp = $self->GetGroup($tag, '0:1');
    my $pi = $self->GetInfo("$grp:PhotometricInterpretation");
    return $Image::ExifTool::Exif::compression{$val};
};
(incomplete, of course, but it already bombs out when you enable the GetInfo line).

I need to use GetInfo, not GetValue (which I think works at this spot) as I need to be able to direct exiftool to the proper tag (i.e., matching group), or do you know another way that enables me to determine the value of an exactly specified tag?
Hayo Baan – Photography
Web: www.hayobaan.nl
Blog: blog.hayobaan.nl
500px: 500px.com/HayoBaan

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 745
  • Professional Photographer and Workshop Leader
    • Hayo Baan – Photography
Re: More detailed JPEG Compression info
« Reply #3 on: January 09, 2017, 09:00:28 AM »
OK, I played a bit more and found a way to get what I want:
Code: [Select]
require Image::ExifTool::Exif;
$Image::ExifTool::Exif::Main{0x103}->{PrintConv} = q{
    my $compression = $Image::ExifTool::Exif::compression{$val};
    if ($val == 7) {
        my $grp = $self->GetGroup($tag, '0:1');
        my ($pi_tag) = grep { /PhotometricInterpretation( \(.*)?/ && $self->GetGroup($_, '0:1') eq $grp } keys($$self{VALUE});
        my ($bps_tag) = grep { /BitsPerSample( \(.*)?/ && $self->GetGroup($_, '0:1') eq $grp } keys($$self{VALUE});
        my $pi = $$self{VALUE}{$pi_tag} // "";
        my $bps = $$self{VALUE}{$bps_tag} // "";
        if ($pi && $bps) {
            if (($pi == 6 && $bps eq "8 8 8") || ($pi == 1 && $bps eq "8")) {
                $compression .= " (Lossy)";
            } else {
                $compression .= " (Lossless)";
            }
        }
    }
    return $compression;
};

This will add (Lossy) or (Lossless) after the JPEG compression type indicator to give more detail on the type of the compression.

The code seems to work well on the files I tested with. I'm not completely certain it is 100% correct in all cases though. For instance an image I scanned with Apples Image Capture as tiff is reported as JPEG compressed too and using the above code as guideline it is reported now as lossless compressed. Could this indeed be true? (I hope so, otherwise there's no benefit in scanning tiff versus jpg).
But this leads me to the question: how can you determine the compression type of a JPG (other than how I do it now)?
« Last Edit: January 09, 2017, 09:10:36 AM by Hayo Baan »
Hayo Baan – Photography
Web: www.hayobaan.nl
Blog: blog.hayobaan.nl
500px: 500px.com/HayoBaan

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 11374
    • ExifTool Home Page
Re: More detailed JPEG Compression info
« Reply #4 on: January 09, 2017, 10:01:52 AM »
Hi Hayo,

I think you probably want to look at only the compression of the main (full-resolution) image.  How about this config file to define a Composite MainImageCompression tag?

- Phil
...where DIR is the name of a directory containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 745
  • Professional Photographer and Workshop Leader
    • Hayo Baan – Photography
Re: More detailed JPEG Compression info
« Reply #5 on: January 09, 2017, 11:37:20 AM »
I think you probably want to look at only the compression of the main (full-resolution) image.  How about this config file to define a Composite MainImageCompression tag?

Neat! Contains a couple of handy features I might want to use at a later stage 8)
But I actually want to know it for all embedded images. Your config file, for instance, does not report the type of JPG compression for the TIFF scan I mentioned, and I'd like to know that too :)

(also: any chance the JPEG compression type can be determined otherwise, I'd like to confirm the JPEG compression in the TIFF is indeed Lossless compressed)
Hayo Baan – Photography
Web: www.hayobaan.nl
Blog: blog.hayobaan.nl
500px: 500px.com/HayoBaan

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 11374
    • ExifTool Home Page
Re: More detailed JPEG Compression info
« Reply #6 on: January 09, 2017, 11:57:04 AM »
Hi Hayo,

OK, here you go.  In this version I generate MyCompression tags for all embedded images and set the family 1 group of the composite tag according to the original IFD.

(also: any chance the JPEG compression type can be determined otherwise, I'd like to confirm the JPEG compression in the TIFF is indeed Lossless compressed)

I haven't had the time to research this, so I can't answer this question right now.

- Phil

Edit: Fixed to always return MyCompression even if other tags not defined.

Edit2: Removed requirement for SubFileType
« Last Edit: January 09, 2017, 01:37:41 PM by Phil Harvey »
...where DIR is the name of a directory containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 745
  • Professional Photographer and Workshop Leader
    • Hayo Baan – Photography
Re: More detailed JPEG Compression info
« Reply #7 on: January 09, 2017, 12:51:47 PM »
Nice, more tricks I can use in future coding!

I needed to remove the requirement for SubFileType to get the config file to report the MyCompression tag for the scanned TIFF (the Desired tags aren't used so they can go too).

The group they are reported in now is e.g. ExifTool:IFD0, not EXIF:IFD0. How do I fix that?
Update: I noticed that with the Compression tag in SubIFD, MyCompression reports it as coming from IFD0…

PS. I guess you prefer this custom tag approach instead of my hack as this way is more future proof?
« Last Edit: January 09, 2017, 01:12:16 PM by Hayo Baan »
Hayo Baan – Photography
Web: www.hayobaan.nl
Blog: blog.hayobaan.nl
500px: 500px.com/HayoBaan

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 11374
    • ExifTool Home Page
Re: More detailed JPEG Compression info
« Reply #8 on: January 09, 2017, 01:33:37 PM »
Hi Hayo,

I needed to remove the requirement for SubFileType to get the config file to report the MyCompression tag for the scanned TIFF (the Desired tags aren't used so they can go too).

The group they are reported in now is e.g. ExifTool:IFD0, not EXIF:IFD0. How do I fix that?

I've updated the attached config file in my last post accordingly.

Quote
Update: I noticed that with the Compression tag in SubIFD, MyCompression reports it as coming from IFD0…

Can you send me the sample (phil at owl.phy.queensu.ca)?

Quote
PS. I guess you prefer this custom tag approach instead of my hack as this way is more future proof?

Yes.  But not completely since we are still accessing some ExifTool internals.

- Phil
...where DIR is the name of a directory containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 745
  • Professional Photographer and Workshop Leader
    • Hayo Baan – Photography
Re: More detailed JPEG Compression info
« Reply #9 on: January 09, 2017, 02:47:29 PM »
Hi Phil,

Keeping you quite busy, sorry :)

I needed to remove the requirement for SubFileType to get the config file to report the MyCompression tag for the scanned TIFF (the Desired tags aren't used so they can go too).

The group they are reported in now is e.g. ExifTool:IFD0, not EXIF:IFD0. How do I fix that?

I've updated the attached config file in my last post accordingly.
This works! (but what if the Compression tag is not in EXIF? Isn't it safer to use GetGroup($key, 0) and use that as the SetGroup? I attached my change.

Quote
Update: I noticed that with the Compression tag in SubIFD, MyCompression reports it as coming from IFD0…

Can you send me the sample (phil at owl.phy.queensu.ca)?
Take for instance the DNG from the DNG preview discussion, but any file that has the compression tag in a SubIFD shows the same problem (e.g. NEFs).
The problem seems to be that (without showing duplicates), a different composite tag (the last instead of the first?) is used instead of the one selected for the non-composite… I'm not sure how easy this is to fix.

Quote
PS. I guess you prefer this custom tag approach instead of my hack as this way is more future proof?
Yes.  But not completely since we are still accessing some ExifTool internals.

Hmm, given all the extra (though for me VERY worthwhile) work we now have gone through, my original quick hack still seems easier though (and almost as maintainable). ;D
Besides, it doesn't introduce a new tag but integrates seamlessly :)

P.S. I tried to understand the TIFF way of handling the JPEG compression by reading the TIFF specs, but I haven't found a definite answer there (probably meaning that the tags alone don't contain the answer). Perhaps the answer lies in the JPEG standard?
Hayo Baan – Photography
Web: www.hayobaan.nl
Blog: blog.hayobaan.nl
500px: 500px.com/HayoBaan

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 11374
    • ExifTool Home Page
Re: More detailed JPEG Compression info
« Reply #10 on: January 09, 2017, 03:16:26 PM »
This works! (but what if the Compression tag is not in EXIF? Isn't it safer to use GetGroup($key, 0) and use that as the SetGroup? I attached my change.

Good.

Quote
The problem seems to be that (without showing duplicates), a different composite tag (the last instead of the first?) is used instead of the one selected for the non-composite… I'm not sure how easy this is to fix.

Right.  Changing the order to match those of the Compression tags is actually fairly straightforward, but setting the group of the final tag is a bit of a problem (that is until ExifTool 10.39 is released).

Quote
Hmm, given all the extra (though for me VERY worthwhile) work we now have gone through, my original quick hack still seems easier though (and almost as maintainable). ;D
Besides, it doesn't introduce a new tag but integrates seamlessly :)

You might be right.  Your way is somewhat simpler.

- Phil
...where DIR is the name of a directory containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 745
  • Professional Photographer and Workshop Leader
    • Hayo Baan – Photography
Re: More detailed JPEG Compression info
« Reply #11 on: January 09, 2017, 03:24:58 PM »
Quote
The problem seems to be that (without showing duplicates), a different composite tag (the last instead of the first?) is used instead of the one selected for the non-composite… I'm not sure how easy this is to fix.

Right.  Changing the order to match those of the Compression tags is actually fairly straightforward, but setting the group of the final tag is a bit of a problem (that is until ExifTool 10.39 is released).
Interesting, will be looking forward to it :)

Quote
Hmm, given all the extra (though for me VERY worthwhile) work we now have gone through, my original quick hack still seems easier though (and almost as maintainable). ;D
Besides, it doesn't introduce a new tag but integrates seamlessly :)

You might be right.  Your way is somewhat simpler.

I guess so (though your code taught me more!). Anyway, as long as I'm not certain that the trick is also applicable to non dngs we can't really use either (for non DNG)
Hayo Baan – Photography
Web: www.hayobaan.nl
Blog: blog.hayobaan.nl
500px: 500px.com/HayoBaan

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 11374
    • ExifTool Home Page
Re: More detailed JPEG Compression info
« Reply #12 on: January 09, 2017, 03:40:04 PM »
Here is a version of the config file that generates the MyCompression tags in the same order as the Compression tags.  The groups won't be set properly for one of the tags if using a version of ExifTool earlier than 10.39.

- Phil
...where DIR is the name of a directory containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 745
  • Professional Photographer and Workshop Leader
    • Hayo Baan – Photography
Re: More detailed JPEG Compression info
« Reply #13 on: January 10, 2017, 03:50:53 AM »
Oh, wow! It works (apart from, as you mentioned, the chosen one showing as “Composite” now – I'm on 10.38 still, of course ;)).

Boy, has this solution grown to be extremely complex! Very helpful for future coding though as it shows a lot of tricks to get things show up where (and how) you want them! 8)
Hayo Baan – Photography
Web: www.hayobaan.nl
Blog: blog.hayobaan.nl
500px: 500px.com/HayoBaan

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 745
  • Professional Photographer and Workshop Leader
    • Hayo Baan – Photography
Re: More detailed JPEG Compression info
« Reply #14 on: January 14, 2017, 07:35:43 AM »
Just to let you know: with 10.39 this works as advertised :)
Hayo Baan – Photography
Web: www.hayobaan.nl
Blog: blog.hayobaan.nl
500px: 500px.com/HayoBaan