Too many photos – deduplicate, categorise and archive for easy access (guide)

11/12/2024
By Ray Shaw
In Featured Guides

Too many photos? Since we went digital from Kodachrome, when you had to pay for film and prints, our 8 billion inhabitants take about 5 billion photos daily.

That is over 5 trillion megabytes (5,000,000,000MB) of storage each year. What is worse, most people admit to taking several of the same shots because digital is free, and they should get at least one good one! Where 3MB 16:9 1080p photos were normal, they are now 5-6MB 4K and HDR.

This is also driving smartphone storage higher—256 GB is the new 128 GB, and depending on the quality, it can hold 60K+ photos.

OK, let’s start to manage too many photos

You cannot do this on your phone, and it is folly to try to do it in Google Photos or iCloud Photos. These cloud apps often compress the image and lose EXIF data. Do not try!

We believe a large monitor and a decent PC/Mac are the minimum requirements. We focus on Windows, but the concepts are similar for MacOS.

At its most basic, it involves using Windows Explorer to move, NOT copy, all photos from the phone (we will talk about digitising real photos in another guide) via a USB cable to the PC. Importantly, this keeps hidden EXIF metadata intact (more later as you need the date, time, location, etc.).

WARNING: If you copy phone photos (cut and paste), the creation date changes as you create a new copy on the PC. If you move, the creation date remains unchanged. Some of the programs below will preserve the creation date and EXIF data.

If you have thousands of photos, buy a 2TB USB-C external SSD like the Kingston 2TB XS1000 USB-C 3.2 Gen 2 external SSD (about $200) or WD My Passport SSD – Is 4TB enough? (about $600) but shop around. USB-C SSDs are not archival but are fast enough to work on and give you lots of swap space (space needed when you move photos).

If you have enough free internal disk space, you can use that, but make sure you have twice as much free space as your photo library needs.

Stage 1 – Move

This section aims to move all photos from the phone using Windows Explorer to an external SSD or hard disk. The end goal is to have new date folders, e.g., all photos from 25 December, in a new PC folder called 241225 (using YY/MM/DD – no /) to allow Explorer to sort these folders by year/month/date).

WARNING: This can be a huge job, so break it down into manageable chunks. Limit moves to 50 photos per date, as larger moves may overwhelm your phone. As you get more confident, you can try larger moves. Do not try to move your entire phone library in one copy operation – chances are it will fail.

Start with a PC (or Mac) and use File Explorer to access the phone. You may need to give the phone permission (Notifications—select file transfer). All Android phone photos are in the DCIM/CAMERA folder.

Select the Explorer option that shows the file name, size and date, not thumbnail images. Click on ‘Modified’ to sort in date order.

Highlight/select photos for one date, then select Cut (Scissors move images)

Go to the SSD, create a YY/MM/DD (no / between numbers) folder and select paste (clipboard icon).

Hint: You can open two Explorers side by side – one for the phone and one for the SSD.

What you have done: Moved photos with EXIF and creation date intact to an external SSD

How to get photos back from Google Photos Cloud

This is complicated and I can’t promise it is correct. When Google Photos syncs (a faux backup), it has the choice of High Quality (compresses file and creates a .json text file of the same name with limited metadata, which it uses for sorting in Photos) or Original Mode (uncompressed with metadata intact).

In either case, you can’t download a ‘copy’ as it changes the file date and strips any metadata. The moral of the story is Google Photos makes a lower-sized copy to save cloud space.

If you want to download photos from Google Cloud, you need to use Google Takeout (which exports the photos in a Zip file) and buy a Windows App called Metadata Fixer that does its best to reinstate the .json data with the jpeg.

You can’t get back all the original data or file size unless you stored it as Original Mode.

Photos on your phone are not compressed and you can use a USB cable to ‘move’ (not copy) to a PC or external SSD.

Stage 2 – Deduplicate – get rid of unnecessary shots

Deduplicate software ranges from freeware that uses date/time and basic image metadata to AI-driven software that looks at images and selects similar ones.

One of the best free deduplicate apps is Alldup (click on English). It uses basic image comparison (select File content) to identify possible duplicates (photos or any file type). You can choose the ones you want removed and delete them. It is recommended for skilled users who understand the options and power behind it. There is a good overview here.

One of the best paid apps is Duplicate Photos Fixer for Windows and Mac. It can also scan Google Drive, iCloud photos, and Dropbox photos and allow you to select the level of similarity. The free trial is limited to 15 photos, and the cost is A$102.95 (there are often online specials).

Stage 3 – Rough Categorising

Let’s assume you have moved all the photos to an SSD and are in different YY/MM/DD (no /) folders. If you have an online calendar, diary or similar, you can start adding relevant information to the folder date. For example, on 241225, it was Christmas, and you can rename the file ‘241225 Xmas’ (we assume you know how to rename folders).

How to finer categorise

There are things you should categorise on:

When/where was it taken? (if you do nothing else, this is the most important)
Who/what is in it?
The occasion

If you don’t have a calendar, start with the most recent folders and look at the photos. Do not be tempted to rename the photos – that is a much later job.

If there are several things on that day (shopping, breakfast food, and dinner food create sub-directories and move these groups to relevant folders. If you want to track photos of pets or people, name these folders: 241202 Fido or grandma, etc. Ensure the names are consistent (start a list of tags) across all folders.

Hidden EXIF data gems to help you sort too many photos

All digital photos have EXIF (Exchangeable Image File Format) metadata, including information about the image’s creation, time/date, GPS location, camera/lens used, and settings.

Some programs can sort metadata to at least organise your photos by date and location (highly recommended to enable GPS location for the phone camera).

A good free program is XnView MP for Windows, MacOS, and Linux. It can read EXIF data, display where it was taken on a map, and more. The interface is a little clunky, but the user guide is here. Its main benefits are showing the date and location on a map and allowing you to rename the photo. Try this first!

Too many photos — GPS coordinates can be shown on a Map in this app.

You may also benefit from Systweak Photo Organizer, which costs A$64.95 and can help you use EXIF metadata, has group renaming, and has more tools.

CyberLink PhotoDirector has basic AI, face recognition (face tagging), and sorting. You can then give faces names, and it tags all photos with that face in them. It is a very fully featured program and perhaps overkill for amateurs. The cost is A$109 for an outright purchase, or you can subscribe monthly.

Caveats

Remember that using Windows Explorer copy will change the photo creation date, making date sorting impossible. But your original shot will still be on the phone with the creation date and all the EXIF details. Use Cut to move and leave the creation date intact.

External SSDs and USB Flash drives are not archival quality—if left unpowered, they will last for a few years. Once you get your albums sorted, consider how long you want to archive these images.

Depending on file size, the best storage option is Cloud (OneDrive, AWS, Google Workspace, iCloud), which locks you into a monthly subscription. The key is to store these photos uncompressed and with EXIF data (do not use compression or storage saver). Remember, the Cloud is simply other people’s computers, and they can fail. Reputable cloud providers use RAID (Redundant Array of Independent disks) to minimise the chances of lost data.

If budget is not an issue, use a RAID Level 5 or 6 NAS (network-attached storage) with at least three or four hard disks.

Remember when burning to a CD was the latest, and we later discovered that cheap disks only have a two-year lifespan and you can’t buy a CD player anymore? There were 100/200/300-year archival standard Delkin Blu-ray BR-D disks, but these seem to have been discontinued. You also need a Blu-ray burner and software.

Please remember that CyberShack cannot accept responsibility for accidentally lost or deleted photos, and our advice is general in nature only.

It is a little premature to ask AI to do it for you

So as not to get your hopes up, there is no nirvana, no software, and nothing to clean up your photos automatically; select the best shots, categorise, and add descriptive meta tags. It is coming and will be among the world’s most popular software.

AI will come to the rescue by reading each photo’s metadata. By combining it with people and object recognition, it may be able to do a first cut and then present you with the draft for approval.

If you want to explore the beginnings of AI photo library management, Excire Foto 2025 (A$352 or $52.90pm) for Mac and Windows may do everything you want. We have not tested this, but it is constantly getting rave reviews. It may replace all the manual steps above.