Author Topic: Massive photo tidyup solution using ExifTool, but what about XMPs?  (Read 4576 times)

refsvik

  • Newbie
  • *
  • Posts: 7
After making a mess for the last 15 years, I finally decided to try do a massive photo cleanup in terms of removing duplicates, renaming files (yyyymmdd_hhmmss.ext) and creating a tidy date-based folder hierarchy.

I started with exporting all my image files from iPhoto, Aperture and Lightroom into a folder structure. There are more than 300K of them. The plan is to remove duplicates (PhotoSweeper seems like a good alternative).

After that, it seems natural to rename files and create the appropriate folder structure and I have found several ExifTool statements to do just that. However - I have not worked out how to rename .XMP and .CR2 files in pairs while moving them to date-based folders without creating XMP orphans, as they do have different creation dates.

The other way could of course be to try to embed the XMP metadata inside the RAW files (if possible) before starting the renaming/moving process.

In any case - I would appreciate any tips on how to do a large image cleanup using Exiftool and other shell script elements.

Sincerely,

Kjell Are Refsvik
Norway

StarGeek

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 2562
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #1 on: December 25, 2015, 10:46:05 PM »
I don't have experience with doing this so I can't directly help, but here's a Google link with multiple examples from this forum.
Troubleshooting hints:
* When posting, include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).
* Double all percent signs (%) in a Windows batch file.
* If your GPS coords are negative, make sure and set the GpsLatitudeRef and GpsLongitudeRef tags correctly.

Phil Harvey

  • ExifTool Author
  • Administrator
  • ExifTool Freak
  • *****
  • Posts: 14879
    • ExifTool Home Page
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #2 on: December 27, 2015, 12:02:29 PM »
Hi Kjell,

This is a bit tricky.  I would first set CreateDate in the XMP to be the same as in the CR2 (I think the CR2 contains a CreateDate, right?):

exiftool -tagsfromfile %d%f.cr2 -createdate -ext xmp DIR

Then rename the files according to CreateDate.  But the problem comes in if there are images shot within the same second.  If this is a possibility, I need to think about this some more.

But embedding the XMP into the CR2 is an alternative, and may make life simpler if the software you use will recognize this XMP.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

refsvik

  • Newbie
  • *
  • Posts: 7
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #3 on: December 27, 2015, 04:10:14 PM »
Thanks Phil. I look forward to any suggestions.

Personally, I hope that exiftool can help me in the endevour to do a massive thermo nuclear cleanup of a vast image collection of images with lots of duplicates and non-consistent folder- and filnames. I am trying to piece together something that is safe to use on a Mac with a minimum of lost data, and that everyone with the same challenge as me could use. I would be very happy for any help to piece together items on the list below or modify the list all together.

- Kjell Are, Norway

----


THERMO NUCLEAR TIDY-UP OF IMAGE FOLDER STRUCTURE

1. Export all images + XMPs from Aperture, iPhoto and Lightroom to a new HD
<manual export in each app>

2. Rename all filenames to lower case:
?

3. Substitute all blank spaces in file- and foldernames with "_" (underscore):
?

4. Ingest XMP metadata into parent image files:
exiftool -r -ext cr2 -ext jpg -ext jpeg -ext tiff -ext tif -ext nef -tagsfromfile %d%f.xmp -xmp

5. Remove all XMPs from disk:
rm -r /Volumes/Bilder3/*.xmp; rm -r /Volumes/Bilder3/*.XMP

6. Remove duplicate images
Run PhotoSweeper GUI to remove dupes and trash them

7. Rename all files with date-based names:
exiftool '-filename<CreateDate' -d %Y%m%d_%H%M%S%%-c.%%le -r -ext cr2 -ext jpg -ext jpeg -ext tiff -ext tif -ext nef - /Volumes/Bilder3

8. Create folder structure with 4-digit year names at root level and folders with YYYYMMDD as level 2:
exiftool ...?

refsvik

  • Newbie
  • *
  • Posts: 7
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #4 on: December 31, 2015, 12:06:07 PM »
I have  been spending some spare time this christmas to extend the workflow on how to do a massive photo cleanup from years of accumulating files and duplicates in iPhoto, Aperture and Lightroom. Here is the current version that still needs a ton of testing and hopefully feedback from you guys and girls:


THERMO NUCLEAR TIDY-UP OF PHOTO FOLDER STRUCTURE
[Provided as is. Proceed to use at your own risk. Not all steps has been completely tested yet, so perform your own testing on copies of files before committing.]

1. Export all files+XMPs from Aperture, iPhoto and Lightroom to a new HD (/Volumes/Bilder3/old):
<manual export from each app>

2. Rename all file- and folder names to lower case:
ls -R | while read upName; do loName=`echo "${upName}" | tr '[:upper:]' '[:lower:]'`; mv "$upName" "$loName"; done

3. Substitute all blank spaces in file- and foldernames with "_" (underscore):
[There are several complete scripts out there, but will need to find the best]

4. Ingest metadata from .xmp files into parent image files:
exiftool -r -ext cr2 -ext jpg -ext jpeg -ext tiff -ext tif -ext nef -tagsfromfile %d%f.xmp -xmp
?: exiftool -ext EXT -tagsfromfile %d%f.xmp -all:all -r Volumes/Bilder3/Old

5. Remove all XMPs from disk:
Verify: find /Volumes/Bilder3/old -name '*.XMP' -type f; find /Volumes/Bilder3/old -name '*.xmp' -type f
Delete: find /Volumes/Bilder3/old -name '*.XMP' -type f -delete; find /Volumes/Bilder3/old -name '*.xmp' -type f -delete

6. Remove duplicate images:
Run PhotoSweeper GUI to remove duplicate photo files and trash them

7. Rename all photo files with date-based names:
exiftool '-filename<CreateDate' -d %Y%m%d_%H%M%S%%-c.%%le -r -ext DNG -ext dng -ext psd -ext PSD -ext cr2 -ext jpg -ext jpeg -ext tiff -ext tif -ext nef - /Volumes/Bilder3/old

8. Move out all video files to a seperate folder
find /Volumes/Bilder3 -name "*.avi" -type f -exec mv {} /Volumes/Bilder3/video \;
find /Volumes/Bilder3 -name "*.mpg" -type f -exec mv {} /Volumes/Bilder3/video \;
find /Volumes/Bilder3 -name "*.mov" -type f -exec mv {} /Volumes/Bilder3/video \;

9. Rename all videofiles with date-based names:
exiftool '-filename<DateTimeOriginal' -d %Y%m%d_%H%M%S%%-c.%%le -r -ext avi -ext AVI -ext mov -ext MOV -ext mpg -ext MPG /Volumes/Bilder3/video

10. Create folder structure with 4-digit year names at root level and folders with YYYYMMDD as level 2:
exiftool '-Directory<CreateDate' -d /Volumes/Bilder3/new/%Y/%Y%m%d -r /Volumes/Bilder3/old

11. Look for files left after cleanup and copy them to a single folder
find /Volumes/Bilder3/old -name "*.*" -type f -exec mv {} /Volumes/Bilder3/unknowns \;

12. Mark panorama photos taken with the Sigma 8mm Lens (lens only used for spherical panorama shots)
[?]
exiftool '-filename<%f_$LensMake.%e' -r /Volumes/Bilder3/old
[Will mark all photos with the lensmake, but how do I do this with only Sigma 8mm images: filename_vr.ext??]

13. Place video files in a single folder
find /Volumes/Bilder3 -name "*.avi" -type f -exec cp {} /Volumes/Bilder3/video \
find /Volumes/Bilder3 -name "*.mpg" -type f -exec cp {} /Volumes/Bilder3/video \
find /Volumes/Bilder3 -name "*.mov" -type f -exec cp {} /Volumes/Bilder3/video \

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 1079
  • Professional Photographer & Software Developer
    • Hayo Baan – Photography
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #5 on: January 01, 2016, 11:54:17 AM »
I actually can't see much wrong with your approach, looks like it does what you want.

Perhaps some steps could be combined/skipped (for instance, why first move the video files into one directory and then move them again to their proper destination using exiftool? You could have exiftool process all video files in one go: exiftool -ext AVI -ext MOV…). You can certainly merge your multiple finds into one at a lot of places, for instance
Code: [Select]
find /Volumes/Bilder3/old -name '*.XMP' -type f; find /Volumes/Bilder3/old -name '*.xmp' -type f can be rewritten as
Code: [Select]
find /Volumes/Bilder3/old -iname '*.xmp' -type f (iname matches case insensitive) and
Code: [Select]
find /Volumes/Bilder3 -name "*.avi" -type f -exec mv {} /Volumes/Bilder3/video \;
find /Volumes/Bilder3 -name "*.mpg" -type f -exec mv {} /Volumes/Bilder3/video \;
find /Volumes/Bilder3 -name "*.mov" -type f -exec mv {} /Volumes/Bilder3/video \;
could become
Code: [Select]
find /Volumes/Bilder3 \( -name "*.avi" -or -name "*.mpg" -or -name "*.mov" \) -type f -exec mv {} /Volumes/Bilder3/video \;
But these are just small things  :D
Hayo Baan – Photography
Web: www.hayobaan.nl

refsvik

  • Newbie
  • *
  • Posts: 7
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #6 on: January 01, 2016, 05:12:36 PM »
Thank you Hayo! I have done some modifications, but would like input on how to rename and move video files with the same exif command.

Here is my current workflow after some more testing and dry runs:


THERMO NUCLEAR TIDY-UP OF PHOTO FOLDER STRUCTURE
This is a workflow, largely using ExifTool to clean up a folder structure of files and XMP sidecar files exported from iPhoto, Aperture and Lightroom and its purpose is to

1. get rid of duplicates
2. enforce a consistent naming scheme of files and folder
3. not loose metadata
3. as well as create a consistent folder hierarchy (YYYY/YYYYmmdd).

There is also the added purpose of filtering out video files to store them separately after the cleanup as they will weigh down a photo archive system

The workflow is provided as is. Proceed to use at your own risk. Not all steps has been completely tested yet, so perform your own testing on copies of files before committing. Keep backups on hand after cleanup to be able to recover from loss of data.

The idea is to rename and move images out of a "old" folder and into a "new" folder where renamed/organized files/folders live. That way, the "old" folder should be almost empty when all these commands have been run. Thanks to Phil Harvey and Hayo Baan for constructive input so far.

- Kjell Are Refsvik, Norway, January 2016

--------------------

1. Export all files with XMP sidecar files from Aperture, iPhoto and Lightroom to a folder on a new HD (/Volumes/Bilder3/old):
<manual export from each app into respective subfolders to avoid collisions between like-named files>

2. Remove duplicate images:
Run PhotoSweeper GUI to remove duplicate photos in "old" folder (and subfolders) and trash them.

3. Substitute all blank spaces in file- and foldernames with "_" (underscore):
[There are several complete scripts out there, but will need to find the best. Not sure I need this step though.]

4. Ingest metadata from .xmp files into parent image while not storing the old version
exiftool -r -ext cr2 -ext jpg -ext jpeg -ext tiff -ext tif -ext nef -ext dng -ext png -tagsfromfile %d%f.xmp -xmp -overwrite_original /Volumes/Bilder3/old
exiftool -r -ext CR2 -ext JPG -ext JPEG -ext TIFF -ext TIF -ext NEF -ext DNG -ext PNG -tagsfromfile %d%f.XMP -xmp -overwrite_original /Volumes/Bilder3/old
[A lot of Warnings when running this: "Warning: Error opening file". What could it be?]

5. Remove all .xmp files from disk:
find /Volumes/Bilder3/old -iname '*.xmp' -type f -delete

6. Rename all file- and folder names to lower case:
ls -R | while read upName; do loName=`echo "${upName}" | tr '[:upper:]' '[:lower:]'`; mv "$upName" "$loName"; done

7. Rename all photo files with date-based names:
1: rename files with CreateDate first: exiftool '-filename<CreateDate' -d %Y%m%d_%H%M%S%%-c.%%le -r -ext dng -ext psd -ext cr2 -ext jpg -ext jpeg -ext tiff -ext tif -ext nef - /Volumes/Bilder3/old
2: rename files with empty CreateDate: exiftool '-filename<DateTimeOriginal' -d %Y%m%d_%H%M%S%%-c.%%le -r -ext dng -ext psd -ext cr2 -ext jpg -ext jpeg -ext tiff -ext tif -ext nef - /Volumes/Bilder3/old
3: rename files with empty CreateDate and DateTimeOriginal: exiftool '-filename<filemodifydate' -d %Y%m%d_%H%M%S%%-c.%%le -r -ext dng -ext psd -ext cr2 -ext jpg -ext jpeg -ext tiff -ext tif -ext nef - /Volumes/Bilder3/old

8. Rename all videofiles with date-based names and move them to a video folder:
exiftool '-filename<CreateDate' -d %Y%m%d_%H%M%S%%-c.%%le -r -ext avi -ext AVI -ext mov -ext MOV -ext mpg -ext MPG -ext m4v -ext M4V -overwrite_original /Volumes/Bilder3/old
find /Volumes/Bilder3/old \( -name "*.avi" -or -name "*.mpg" -or -name "*.mov" -or -name "*.m4v" \) -type f -exec mv {} /Volumes/Bilder3/video \;

9. Create folder structure with 4-digit year names at root level and folders with YYYYMMDD as level 2:
exiftool '-Directory<CreateDate' -d /Volumes/Bilder3/new/%Y/%Y%m%d -r /Volumes/Bilder3/old
exiftool '-Directory<DateTimeOriginal' -d /Volumes/Bilder3/new/%Y/%Y%m%d -r /Volumes/Bilder3/old
exiftool '-Directory<filemodifydate' -d /Volumes/Bilder3/new/%Y/%Y%m%d -r /Volumes/Bilder3/old

10. Look for files left after cleanup and copy them to a single folder
find /Volumes/Bilder3/old -name "*.*" -type f -exec mv {} /Volumes/Bilder3/unknowns \;

11. Append "_vr" to filenames taken with the Sigma 8mm lens (this lens was/is only used for spherical panoramas)
[?]

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 1079
  • Professional Photographer & Software Developer
    • Hayo Baan – Photography
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #7 on: January 02, 2016, 10:01:17 AM »
Hi Kjell,

Step 4:
I can see two errors here, first of all the -ext option is case insensitive so you are basically running the same command over each file, twice (why don't you try it with --ext xmp? This will run exiftool on anything but the xmp files).
Note that not all files may have a matching .xmp file and this is likely the cause of the file opening warnings.

To rename files in one go, simply repeat the assignments in order of importance (most important last), tags that don't exist won't overwrite the earlier setting. And, instead of listing all extensions, why don't you just have exiftool run on all files? (by the way, you had an extra - in front of the directory name, which I think was wrong).
Step 7 and the first half of 8 could then become something like this exiftool '-filename<filemodifydate' '-filename<DateTimeOriginal' '-filename<CreateDate' -d %Y%m%d_%H%M%S%%-c.%%le -r /Volumes/Bilder3/old
(are you absolutely sure about the %%-c and %%le though? it produces just %-c and %le for me…)

If you are unsure about non-image/movie files in your directory structure, this will list them:
find -type f -not \( -iname '*.cr2' -or -iname '*.[dp]ng' -or -iname '*.psd' -or -iname '*.nef' -or -iname '*.jpg' -or -iname '*.jpeg' -or -iname '*.tif*' -or -iname '*.mov' -or -iname '*.avi' -or -iname '*.mpg' -or -iname '*.m4v' -or -iname '*.xmp' \)
Hayo Baan – Photography
Web: www.hayobaan.nl

refsvik

  • Newbie
  • *
  • Posts: 7
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #8 on: January 02, 2016, 11:35:18 AM »
Thank you again, Hayo. That really tightens things up. When it comes to the %%-c and %%le issue, I got the tip here, and have no problem with it. http://ninedegreesbelow.com/photography/exiftool-commands.html
I will however test it according to your suggestions.

Here is a new version, tested, modified and adjusted and are getting closer to something I can use. I did rethink the desire to move out special types of content, for instance the panoramic files (taken with my Sigma 8mm). Those will now be renamed and stay with the other files in the main hierarchy. The filetypes I do want to take out are the video files that end up in my image archive. The screenshots should (as images) be allowed to stay within the image archive and tagges as such.

So from both a metadata, archiving and exiftool-tech perspective, I would love to get your further feedback on this workflow to be able to fine-tune it even more before committing to a full cleanup of all my images.

-Kjell Are


THERMO NUCLEAR TIDY-UP OF PHOTO FOLDER STRUCTURE
After many many years of archiving pictures (I have 300K+ in my Lightroom archive now), changing software and doubling up on many files when moving to a new archive system, I have a lot of duplicates and junk in my image collection. The purpose of this workflow is to clean that up and start fresh. This is a workflow uses largely using ExifTool to clean up a folder structure of files and XMP sidecar files exported from iPhoto, Aperture and Lightroom and its purpose is to :

1. get rid of duplicates
2. change filenames into a consistent one for files and folders
3. not loose metadata (or as little as possible)
4. create a consistent folder hierarchy (YYYY/YYYYmmdd/YYYYddmm_hhmmss.ext)
5. and filter out (and store separately) files like screenshots and videos

The workflow is provided as is and is developed for Mac OS X in mind. Proceed to use at your own risk. Not all steps has been completely tested yet, so perform your own testing on copies of files before committing. Keep backups on hand after cleanup to be able to recover from loss of data. The idea of this workflow is to rename and move images out of a "old" folder and into a "new" folder where renamed/organized files/folders live. That way, the "old" folder should be almost empty when all these commands have been run. Make sure you understand that this script WILL modify your RAW files, both in terms of metadata content and filename. Thanks to Phil Harvey and Hayo Baan for constructive input so far. I would like you to share your experience or modifications to this at http://u88.n24.queensu.ca/exiftool/forum/.

- Kjell Are Refsvik, Norway, January 2016

--------------------

1. Export all files with XMP sidecar files from Aperture, iPhoto and Lightroom to a folder on a new HD (/Volumes/Bilder3/old):
<manual export from each app into respective subfolders to avoid collisions between like-named files>


2. Remove duplicate images:
Run PhotoSweeper (http://overmacs.com/) to remove duplicate photos in "old" folder (and subfolders) and trash them.
[This will presumably leave a lot of orphaned .XMP sidecar files, but does it matter?]


3. Substitute all blank spaces in file- and foldernames with "_" (underscore):
[There are several complete scripts out there, but will need to find the best. Not sure I need this step though?]


4. Ingest metadata from .xmp files into parent image while not storing the old version
exiftool -r --ext xmp -tagsfromfile %d%f.xmp -xmp -overwrite_original /Volumes/Bilder3/old


5. Remove all .xmp files from disk:
verify: find /Volumes/Bilder3/old -iname '*.xmp' -type f
delete: find /Volumes/Bilder3/old -iname '*.xmp' -type f -delete


6. Rename all videofiles with date-based names and move them to a "video" folder:
mkdir /Volumes/Bilder3/video; exiftool '-filename<filemodifydate' '-filename<DateTimeOriginal' '-filename<CreateDate' -d /Volumes/Bilder3/video/%Y/%Y%m%d_%H%M%S%%-c.%%le -r -ext avi -ext mov -ext mpg -ext m4v -ext mts -overwrite_original /Volumes/Bilder3/old


7. Rename all screenshot files (osx/ios) with date-based names, move them to the "new" folder and add a "screenshots" postfix:
exiftool '-filename<filemodifydate' '-filename<DateTimeOriginal' '-filename<CreateDate' -d /Volumes/Bilder3/new/%Y/%Y%m%d_%H%M%S%%-c.%%le -ext png -r /Volumes/Bilder3/old


8. Rename all spherical panorama files with date-based names, move them to the "new" folder and add a "panorama" postfix:
exiftool '-filename<filemodifydate' '-filename<DateTimeOriginal' '-filename<CreateDate' -d /Volumes/Bilder3/new/%Y/%Y%m%d/%Y%m%d_%H%M%S%%-c_panorama.%%le -if '$focallength eq '8.0 mm'' -r /Volumes/Bilder3/old


9. Rename and move all photo files with date-based names (Those with CreateDate metadata intact first, secondly those with DateTimeOriginal intact and thirdly those with only FileModifyDate set):
exiftool '-filename<filemodifydate' '-filename<DateTimeOriginal' '-filename<CreateDate' -d /Volumes/Bilder3/new/%Y/%Y%m%d/%Y%m%d_%H%M%S%%-c.%%le -r -ext dng -ext psd -ext cr2 -ext jpg -ext jpeg -ext tiff -ext tif -ext nef /Volumes/Bilder3/old


10. Look for files left after cleanup and copy them to a single folder
mkdir /Volumes/Bilder3/new/unknowns; find /Volumes/Bilder3/old -name "*.*" -type f -exec mv {} /Volumes/Bilder3/unknowns \;

Hayo Baan

  • Global Moderator
  • ExifTool Freak
  • *****
  • Posts: 1079
  • Professional Photographer & Software Developer
    • Hayo Baan – Photography
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #9 on: January 02, 2016, 01:10:59 PM »
When it comes to the %%-c and %%le issue, I got the tip here, and have no problem with it. http://ninedegreesbelow.com/photography/exiftool-commands.html

My bad, I tested incorrectly (I didn't use it to actually rename the file but used it just in combination with showing a date, e.g.,  exiftool -d %Y%m%d_%H%M%S%%-c.%%le test.jpg and then %%c and %%e make no sense…

10. Look for files left after cleanup and copy them to a single folder
mkdir /Volumes/Bilder3/new/unknowns; find /Volumes/Bilder3/old -name "*.*" -type f -exec mv {} /Volumes/Bilder3/unknowns \;

Here you could actually leave out -name "*.*".

Other than that, it looks well. Be sure to test rigorously though! And capture all the command output in e.g., a log file so you can look through it again.
Hayo Baan – Photography
Web: www.hayobaan.nl

gamin

  • Jr. Member
  • **
  • Posts: 22
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #10 on: January 02, 2016, 02:11:43 PM »
Hi,

I've written a Perl/ExifTool script to do this. To summarize:

1. It renames all picture and video files like this: YYYY-MM-DD_HH-MM-SS.ext
2. It optionally recurses through directories. On my computer, it takes 23 minutes to go through some 40,000 images and some 15,000 xmp/thm files.
3. It handles multiple pictures in the same second by appending a -c sequence number, taking into account the date/time/millisecond fields (if available), the internal filenumber (if available) and the original filename to resolve the original order.
4. It handles filename collisions when renaming (e.g. if files are named picture-01.jpg and picture-02.jpg, but the -02 file is actually earlier than the -01 file).
5. It can write a CSV log of the result.
6. It can run in "preview" mode, and write a CSV log of the expected result.
7. The log file can be manually edited and then read back by the script. By editing the proper columns, the user can prevent renaming of individual files or directories, or force the renaming to any string of the user choice (name collisions are handled when running the script).
8. It renames xmp and thm files according to the renaming of the "real image" file.
9. It can optionally compute an MD5 hash, which can be used to find duplicates (along with the new filename, of course).

I'm finalizing the automated tests, and I have only limited documentation, but in a week or so I could make it available to a couple of testers.

Martin

refsvik

  • Newbie
  • *
  • Posts: 7
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #11 on: January 04, 2016, 12:17:43 PM »
Martin: Would love it if you can share it when ready. Would like to see the finer details of this.

Hayo: Glad we agree ;-) Also, thank you for the find-correction.

I have yet done more testing and have taken 100.000 files for a spin and it seems to work fine. Still more testing is needed to verify that the metadata ar not lost along the way, and that files still work. Also - I added a new point at the end, to try to figure out how to go looking for a piece of metadata that could be a parent folder postfix to make the structure more humanly readably.

Kjell Are

Extreme tidy-up of digital photo collection

After many many years of archiving pictures (I have 300K+ in my Lightroom archive now), changing software and doubling up on many files when moving to a new archive system, I have a lot of duplicates and junk in my image collection. The purpose of this workflow is to clean that up and start fresh. This is a workflow uses largely using ExifTool to clean up a folder structure of files and XMP sidecar files exported from iPhoto, Aperture and Lightroom and its purpose is to :

1. get rid of duplicates
2. change filenames into a consistent one for files and folders
3. not loose metadata (or as little as possible)
4. create a consistent folder hierarchy (YYYY/YYYYmmdd/YYYYddmm_hhmmss.ext)
5. and filter out (and store separately) files like screenshots and videos

The workflow is provided as is and is developed for Mac OS X in mind. Proceed to use at your own risk. Not all steps has been completely tested yet, so perform your own testing on copies of files before committing. Keep backups on hand after cleanup to be able to recover from loss of data. The idea of this workflow is to rename and move images out of a "old" folder and into a "new" folder where renamed/organized files/folders live. That way, the "old" folder should be almost empty when all these commands have been run. Make sure you understand that this script WILL modify your RAW files, both in terms of metadata content and filename. Thanks to Phil Harvey and Hayo Baan for constructive input so far. I would like you to share your experience or modifications to this at http://u88.n24.queensu.ca/exiftool/forum/.

A similar approach to cleaning up a image library can be found here: http://fortes.com/2015/command-line-photo-organization/

- Kjell Are Refsvik, Norway, January 2016

--------------------

1. Export all files with XMP sidecar files from Aperture, iPhoto and Lightroom to a folder on a new HD (/Volumes/Bilder3/old):
<manual export from each app into respective subfolders to avoid collisions between like-named files>


2. Remove duplicate images:
Run PhotoSweeper (http://overmacs.com/) to remove duplicate photos in "old" folder (and subfolders) and trash them.
[This will presumably leave a lot of orphaned .XMP sidecar files, but does it matter?]
Also, could be done with fdupes and the command $fdupes -R -d -N (deletes duplicate photos that are identical)


3. Ingest metadata from .xmp files into parent image while overwriting the original file (only to be done if you have a backup of original files)
exiftool -r --ext xmp -tagsfromfile %d%f.xmp -xmp -overwrite_original /Volumes/Bilder3/old


4. Remove all .xmp sidecar files from disk:
verify: find /Volumes/Bilder3/old -iname '*.xmp' -type f
delete: find /Volumes/Bilder3/old -iname '*.xmp' -type f -delete


5. Rename all videofiles with date-based names and move them to a "video" folder:
mkdir /Volumes/Bilder3/video; exiftool '-filename<filemodifydate' '-filename<DateTimeOriginal' '-filename<CreateDate' -d /Volumes/Bilder3/video/%Y/%Y%m%d_%H%M%S%%-c.%%le -r -ext avi -ext mov -ext mpg -ext m4v -ext mts -ext mp4 -ext 3gp -overwrite_original /Volumes/Bilder3/old


6. Rename all screenshot files (osx/ios) with date-based names, move them to the "new" folder and add a "screenshots" postfix to its name:
mkdir /Volumes/Bilder3/new; exiftool '-filename<filemodifydate' '-filename<DateTimeOriginal' '-filename<CreateDate' -d /Volumes/Bilder3/new/%Y/%Y%m%d_%H%M%S%%-c_screenshot.%%le -ext png -r /Volumes/Bilder3/old


7. Rename all spherical panorama files (taken with the Sigma 8mm EX lens) with date-based names, move them to the "new" folder and add a "panorama" postfix to its name:
exiftool '-filename<filemodifydate' '-filename<DateTimeOriginal' '-filename<CreateDate' -d /Volumes/Bilder3/new/%Y/%Y%m%d/%Y%m%d_%H%M%S%%-c_panorama.%%le -if '$lens eq "8.0 mm"' -r /Volumes/Bilder3/old


8. Rename and move all general photo files with date-based names (those with CreateDate metadata intact first, secondly those with DateTimeOriginal intact and thirdly those with only FileModifyDate set):
exiftool '-filename<filemodifydate' '-filename<DateTimeOriginal' '-filename<CreateDate' -d /Volumes/Bilder3/new/%Y/%Y%m%d/%Y%m%d_%H%M%S%%-c.%%le -r -ext dng -ext psd -ext cr2 -ext jpg -ext jpeg -ext tiff -ext tif -ext nef -ext crw /Volumes/Bilder3/old


9. Look for any files left after cleanup and copy them to a single folder
mkdir /Volumes/Bilder3/new/unknowns; find /Volumes/Bilder3/old -type f -exec mv {} /Volumes/Bilder3/new/unknowns \;

10. Traverse all image files in the end to search for relevant metadata to add to the parent folder name. That way, the hierarchy becomes humanly readable when importing it as a Lightroom Catalog. Would preferably be a name of an event or a location.

[Create a loop where all files are traversed to find usable metadata in order to create human readable folder names, e.g. /Volumes/Bilder3/2004/20040710_lofoten]
[But how...?]



gamin

  • Jr. Member
  • **
  • Posts: 22
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #12 on: January 04, 2016, 02:17:16 PM »
Hi Kjell,

Getting accurate dates from video files is sometimes complicated, as you probably know. The script I've written uses the following heuristics:

1. Some video files have an internal DateTimeOriginal.
2. Some video files have a "companion" thm file, which is actually a JPEG file and this JPEG file usually has a DateTimeOriginal.
3. When all else fails, use the CreateDate or the FileModifyDate, depending on file formats and availability of thm file.

I'll email you the script soon, so you can see the details.

Martin

refsvik

  • Newbie
  • *
  • Posts: 7
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #13 on: January 05, 2016, 07:28:46 AM »
Yes, dates can be a tricky matter. Love to read more about your script. Just been through a 300.000 batch test of files and my current script just found 596 video files hiding in that structure. I just looked manually through around 50 of the oldest ones (from around 2000), and cannot find any with the wrong date across 2 different Canon IXUS cameras, using my current script. I will keep testing though, and look forward to more details from you.

Best regards,

Kjell Are

gamin

  • Jr. Member
  • **
  • Posts: 22
Re: Massive photo tidyup solution using ExifTool, but what about XMPs?
« Reply #14 on: January 10, 2016, 06:08:32 PM »
Hi Kjell,

I have a version of my script you can try. Email me or send me a private message through the forum so I can send you the address where you can download it.
If anyone else is interested in testing this, please let me know.

The documentation is attached.

Martin