Author Topic: Question to NoDups  (Read 1946 times)

herb

  • Sr. Member
  • ****
  • Posts: 298
Question to NoDups
« on: July 11, 2018, 10:11:12 AM »
Hello Phil,

in order to avoid duplicate entries in listtype tags I did some tests with NoDups feature.
In forum I have seen that all NoDups-examples use separator ##.
Code: [Select]
exiftool -sep "##" "-keywords<${keywords;NoDups}" DirOrFileDuring my tests I have seen that using e.g. separator ? or | will lead to unexpected results.

Ok, ## is a good separator, but it cannot be sent when -stay_open is used.

So my question is: Which characters must not be used as separator for NoDups feature.

Thanks for your help in advance.

Best regards
Herb

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 15115
    • ExifTool Home Page
Re: Question to NoDups
« Reply #1 on: July 13, 2018, 07:17:28 AM »
Hi Herb,

Good point.  I'll fix this in version 11.07.  I should have been quoting these special characters in the regular expression.

Thanks for pointing this out.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 15115
    • ExifTool Home Page
Re: Question to NoDups
« Reply #2 on: July 13, 2018, 07:53:49 AM »
Hi again,

This also raises the issue of a big oversight on my part.  You should be able to start an argument with "#' in the argfile.  ExifTool 11.07 will allow you to add "##" as an argument like this:

#[NOT_COMMENT]##

So if you have any argument that starts with "#", just replace this with "#[NOT_COMMENT]#" in the argfile to avoid it being interpreted as a comment.


- Phil

Edit: I just realized the you can in fact add an argument of "##" by putting a space before it.  But then there is no mechanism to have an argument start with a space.  I will re-think this.
« Last Edit: July 13, 2018, 08:07:29 AM by Phil Harvey »
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 15115
    • ExifTool Home Page
Re: Question to NoDups
« Reply #3 on: July 13, 2018, 08:27:46 AM »
OK, second try.

ExifTool 11.07 will add a feature allowing an argfile line to begin with "#[CSTR]".  The rest of the line is taken as a standard C string.  As well as allowing spaces at the start of a line, this new feature also allows newlines to be embedded.  For example:

#[CSTR] this argument has a leading space

#[CSTR]this\nargument\nhas\nmultiple\lines

This also provides a mechanism to have an empty argument if necessary (which wasn't allowed before):

#[CSTR]

As well as allowing "##" as an argument, which was the initial impetus:

#[CSTR]##

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

herb

  • Sr. Member
  • ****
  • Posts: 298
Re: Question to NoDups
« Reply #4 on: July 27, 2018, 11:58:12 AM »
Hello Phil,

thanks for the new version 11.07 and also many thanks for all the corrections.

A very short test showed that e.g. character "," is working as separator for NoDups feature.
The new feature #[CSTR] is also working properly when I start Exiftool from Dos-Box and call an argument file with -@ option.

But when I send a line starting with #[CSTR] via pipe to Exiftool - using the -stay_open feature -  this line is still ignored by Exiftool.
Or is there a misunderstanding ony my side?

Best regards
Herb

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 15115
    • ExifTool Home Page
Re: Question to NoDups
« Reply #5 on: July 27, 2018, 12:05:27 PM »
Hi Herb,

Ooops, you're right.  I have different code to filter arguments from the stay_open file.  The new feature needs to be applied in two different places.  :(

I'll fix this in the next release.

Thanks for letting me know.

- Phil
« Last Edit: July 27, 2018, 12:15:59 PM by Phil Harvey »
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

herb

  • Sr. Member
  • ****
  • Posts: 298
Re: Question to NoDups
« Reply #6 on: July 28, 2018, 04:41:43 AM »
Hello Phil,

thanks for your quick reply.

You created #[CSTR] feature and now we have a very easy possibility to enter e.g. NEWLINE into a textstring.

Please allow to think loud about the output of such a NEWLINE.
As far as I know Exiftool changes - in general - all binary characters to a dot inside the output.
Would it be helpful to modiy this feature and change NEWLINE to "\n" instead of a dot?

Best regards
Herb

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 15115
    • ExifTool Home Page
Re: Question to NoDups
« Reply #7 on: July 28, 2018, 07:52:45 AM »
Hi Herb,

A change like that would require a new command-line option (otherwise it wouldn't be backward compatible), but I really try to avoid that if possible.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

herb

  • Sr. Member
  • ****
  • Posts: 298
Re: Question to NoDups
« Reply #8 on: July 28, 2018, 10:24:21 AM »
Hello Phil,

thanks for your reply.
I agree Exiftool should always be backward compatible and I also agree not to introduce a new commandline option.
But what do you think about using an userParam - called NEWLINE?

Best regards
Herb
 

StarGeek

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 2748
Re: Question to NoDups
« Reply #9 on: July 28, 2018, 01:38:20 PM »
Would it be helpful to modiy this feature and change NEWLINE to "\n" instead of a dot?

This can be done without adding new command options.  Add the following subroutine to your .exiftool_config file, somewhere before the main definitions (i.e. before %Image::ExifTool…)
Code: [Select]
# Replace NewLines (and other characters) with their escape sequences
sub RepNL {
($_)=@_;
my %replace = (
"\n" => "\\n", # line feed 0x0A
"\r" => "\\r", # carriage return 0x0D
"\t" => "\\t", # tab 0x09
);
s/(@{[join '|', map { quotemeta($_) } keys %replace]})/$replace{$1}/g;
}

You can now use the RepNL (rename as you desire for ease of use) as an advanced formatting option. 

Examples:
Code: [Select]
C:\>exiftool -Echo "Set Description and Caption with whitespace control characters" -P -overwrite_original -E -description="NL:&#x0a; CR:&#x0d; Tab:&#x09;" -Caption-Abstract="NL:&#x0a; CR:&#x0d; Tab:&#x09;" -E y:\!temp\Test3.jpg
Set Description and Caption with whitespace control characters
    1 image files updated

C:\>exiftool -g1 -a -s -Description -Caption-Abstract y:\!temp\Test3.jpg
---- XMP-dc ----
Description                     : NL:. CR:. Tab:.
---- IPTC ----
Caption-Abstract                : NL:. CR:. Tab:.

C:\>exiftool -g1 -a -s -echo "Global Replace NLs" -api "Filter=RepNL($_)" -Description -Caption-Abstract y:\!temp\Test3.jpg
Global Replace NLs
---- XMP-dc ----
Description                     : NL:\n CR:\r Tab:\t
---- IPTC ----
Caption-Abstract                : NL:\n CR:\r Tab:\t

C:\>exiftool -g1 -a -s -echo "Using -p, selectively replace"  -p "${Caption-Abstract;RepNL($_)}" y:\!temp\Test3.jpg
Using -p, selectively replace
NL:\n CR:\r Tab:\t

Other non-printable characters can be added if you desire.
Troubleshooting hints:
* When posting, include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).
* Double all percent signs (%) in a Windows batch file.
* If your GPS coords are negative, make sure and set the GpsLatitudeRef and GpsLongitudeRef tags correctly.

herb

  • Sr. Member
  • ****
  • Posts: 298
Re: Question to NoDups
« Reply #10 on: July 29, 2018, 04:10:31 AM »
Hello StarGeek, hello Phil,

@StarGeek:
Thanks for your hints and thanks for your Perl code; it could be very helpful.
But your design will only work when I ask explicitely for a tag; e.g.: -Description in your example.
Or am I wrong?
Will it also work when I display all tags with -all:all because of the -api filter=... option?

Edit:
In meantime I did some more tests and now I also got it working with filter defined inside config file.


@Phil:
Please allow some additional questions to #[CSTR] feature:
(1) I thought \n is system depending: so \n == h'0A0D on Windows systems.
     A short test showed that also on Windows only h'0A will be used.

     Edit: Please forget (1): I mixed NEWLINE and LINEFEED.
(2) Using #[CSTR] feature I also think on Unicode characters, strings as values of listtype tags and of course also on structures.
     In a short test I entered Chinese characters to IPTC and XMP listtype tags and I have seen no restriction.
     But just to be sure: are there any restrictions?
     In combination with structures I think on possible problems because of escaping some characters.

Thanks again for your help in advance.
Best regards
Herb
« Last Edit: July 29, 2018, 08:28:50 AM by herb »

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 15115
    • ExifTool Home Page
Re: Question to NoDups
« Reply #11 on: July 30, 2018, 07:20:25 AM »
Very smart StarGeek.

The #[CSTR] should work fine for inputting structures and Unicode characters.  The only difference is that it unescapes characters as if it was a double-quoted Perl string (no not actually a C string because Perl adds a few extra features).  See section 2.3.2 here.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

herb

  • Sr. Member
  • ****
  • Posts: 298
Re: Question to NoDups
« Reply #12 on: July 30, 2018, 11:55:21 AM »
Hello Phil, hello StarGeek,

thanks again for your detail hints.

I see, I have to learn more about Perl.
But at the moment it looks like that I have only to escape backslash.

Best regards
Herb

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 15115
    • ExifTool Home Page
Re: Question to NoDups
« Reply #13 on: July 30, 2018, 12:32:01 PM »
Hi Herb,

That reminds me.  (For now) you must also escape "$" and "@" symbols.  I'll fix this in the next release.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

herb

  • Sr. Member
  • ****
  • Posts: 298
Re: Question to NoDups
« Reply #14 on: July 30, 2018, 01:54:40 PM »
Hello Phil,

thanks; you are really great!

Best regards
Herb