Advice for huge project: Import, file rename, keywords...

Hi all,

I've spent the last year in Africa filming and have 18TB of footage to edit and produce youtube content from. This was mostly shot on a Sony A6600, but also with 2 Osmo Actions, 1 Osmo Pocket and a Mavic Air, all at 4K 25P.
Sony files are .MP4 with a .XML file that FCPS won't import, the rest are all .MP4 with a few .MOV for good luck.

Now safely home I've managed to back it all up in 3 or 4 places and have a (double backed up) T3 32TB RAID 0 enclosure to edit from. Due to slightly challenging conditions in existing, filming and backing up the footage whilst living in a vehicle, the file structure whilst backing up isn't great (ad hoc is being kind) and default file names used from the cameras, date and time not always correct. I have 64 files called C0001.mp4 for example

In the end it will breakdown into about 50 or so, 20-40 minute episodes. My thoughts are to create a new library for each one, leave footage in place on the T3 enclosure whilst editing, at the end export a library with media included, duplication there, but it's from unstructured to finished. 

I also have about 1,000 raw images in Lightroom to import and use.

I'm ok with the very basics of FCPX, but realise that researching options now will likely save me a lot of pain in the future, so any advice appreciated please, its taken 15 days just copying  and backing up as I could only get 4 or 5TB USB2 external disks in Africa. (Beyond Compare was invaluable in ensuring I got everything).

What I'd like to achieve is something like done on this video, everything keyworded and easily searchable through the Event Browser, as clips will often be used in multiple episodes, months apart in the editing process.


I'd appreciate advice on:
1. Is there any software that allows batch renaming +and+ very easy skimming of a clip +and+ adding of keywords/notes that FCPX will then import successfully?
This saves viewing clips twice, else I will need to skim to give it a name, then skim later in FCP to give it key words.
I'd like a file name of date_time_xxxxx with the x's words I choose based on the clips content.

2. I have a 2019 MBP with 32Gb, 4TB SSD and 8GB Graphics hooked up to a T3 enclosure with RAID 0 32TB on 7200rpm 3.5" disks. Could I create 1 Library, import all 18TB into it (leaving the files in place!!) on the T3 and catalogue all of the files in there? Sounds crazy, I'd imagine it would crash on trying, but it would be nice to have 1 Library that I open, search in, find clips and drag to the Library I'm editing on.

And any general advice appreciated if you feel different approaches would be better etc, thanks!

OK, quite a few hours later and it seems its ok to import everything into FCP if I break it down into 2TB chunks or so. Rename the folder to what you'd like the keyword collection to be called, drag in the folder, walla....
Then use batchclipApply Custom Name renaming and add in keywords.

500Gb of clips imported but left in place on the TB3 disks only created a 226MB Library file on the internal disk, so thats fine, seems quite responsive in the browser too...

You can batch rename and apply keywords (tags) in the Finder, as well as use a folder structure for keywords. All of it translates into FCP.

honeybadger wrote: ...have 18TB of footage to edit and produce youtube content from. This was mostly shot on a Sony A6600, but also with 2 Osmo Actions, 1 Osmo Pocket and a Mavic Air, all at 4K 25P.
Sony files are .MP4 with a .XML file that FCPS won't import, the rest are all .MP4 with a few .MOV for good luck... have 64 files called C0001.mp4 for example...My thoughts are to create a new library for each one, leave footage in place on the T3 enclosure whilst editing, at the end export a library with media included, duplication there, but it's from unstructured to finished...
1. Is there any software that allows batch renaming +and+ very easy skimming of a clip +and+ adding of keywords/notes that FCPX will then import successfully?..This saves viewing clips twice...2. I have a 2019 MBP with 32Gb, 4TB SSD and 8GB Graphics hooked up to a T3 enclosure with RAID 0 32TB on 7200rpm 3.5" disks. Could I create 1 Library, import all 18TB into it (leaving the files in place!!) on the T3 and catalogue all of the files in there? Sounds crazy, I'd imagine it would crash on trying, but it would be nice to have 1 Library that I open, search in, find clips and drag to the Library I'm editing on....

I've worked on documentaries roughly that size -- it is a major undertaking. Ideally this scope of project would have a dedicated 1st Assistant Editor who is an expert at issues involving data management, NLE performance, and pre-edit organization. That's not always possible but your media in terms of camera hours is roughly similar to a major feature film. Some suggestions:

- Verify all files are in folders by shooting day and camera name

- Consider de-duplicating all media files. Typically on projects like this some content will be accidentally duplicated, which causes problems later. I use Gemini 2 for this: apps.apple.com/us/app/gemini-2-the-dupli...r/id1090488118?mt=12

- Before ingest, rename all media files to be globally unique by adding a 5-digit incrementing serial number. This is to avoid various FCP behaviors involving relink, XML import and to generally streamline post-production file references. This can be done with Finder or specific rename tools:


- In general the best strategy is deferring most organization, curating and logging to within FCP. No other tool is that fast at skimming and rating media.

- The media from the A6600 data should ingest. However Sony Alpha cameras use tree-oriented media which prevents ingest with "leave files in place". The two solutions are (1) Re-wrap with EditReady2 or (2) Copy the .MP4 files out of the folder and ingest. That is generally not a good idea but with Sony XAVC-S files it usually works OK.

- 4k Sony XAVC-S and DJI files are sluggish to edit, even on the newest Mac. This is magnified by having 100+ camera hours of material which needs curating. It is likely you will require proxies for this. You can do your own tests to see if that's necessary. If proxies are required it could take several days to a week to generate those for this amount of data. Those should be placed in a separate folder you define with the library storage manager.

The space required for proxies (which by default are in ProRes Proxy encoding) is roughly 60% the size of 8-bit 4:2:0 H264. By default proxies are 1/2 the linear and 1/4 the pixel resolution of the originals, so for UHD 4k, proxies would be 1080p. As of 10.4.9 FCP has H264 and alternate size proxies. See the Ripple Training class on 10.4.9 media management: www.rippletraining.com/products/final-cu...ent-in-fcp-x-10-4-9/

- After ingest, skim and spot-check the files and ensure they are sorted in chronological order. If any cameras were not set to correct time of day, you can batch update the dates from within FCP. Select all wrongly-dated clips from that device and batch update the date/time with Modify>Adjust Content Created Date and Time.

- For some possible keywording strategies, see this post:  www.fcp.co/forum/4-final-cut-pro-x-fcpx/...w-suggestions#108930 [\url]

- Make sure all media is imported using only "leave files in place", that the FCP cache is stored externally and that any proxies are also stored external to the library. Ideally place the remaining "lean library" and cache on a separate dedicated SSD.

- Re using one or multiple libraries, if the 18 TB is all or mostly 4k H264, that equates to over 200 camera hours of material, possibly 8,000+ clips. That is a lot of data. It probably helps FCP's internal database to spread that across several events, since each event has its own separate SQLite database. However do not create dozens of events -- they are not designed to be used that way. Rather use mostly keyword ranges for organization.

- The main initial organizational methods should be favorites, rejects and keyword ranges, not string-outs on multiple timelines. It is faster to query that and insert to the final timeline than to copy data between timelines.

- Before ingesting a huge amount of data, you might pick a subset such as 5-10% of it, ingest that as a test, see how it behaves regarding performance, space consumption for proxies, etc. Verify the various data components such as proxies, cache, etc. are being properly located. Skim that and think about keyword strategies.

- Make sure the drive containing the library is backed up regularly. In Finder periodically right-click the library and select "duplicate". Each time you do that, don't delete the prior one, just let it increment the suffix number. That provides an additional layer of backup versions if the library somehow gets damaged. Make sure those are backed up daily. A properly configured "lean library" is not that big -- usually no more than 100 MB.

Joema, and others, thank you so much for the considered, knowledgable and extremely useful replies, it helped to calm my nerves a bit.Busy but productive week, photos are all in LR and catalogued, video and photos all backed up at least 3 times with a nightly sync as a minimum.1— Duplication was a good place to start as suggested, backing up full SDHC cards in the African bush in very difficult climates with limited electricity leads to constant brain fades, so I had been ultra cautious against this and managed to find about 900Gb of duplications, now removed.The best tools I found for this were 1. Duplicate File Finder which was always accurate, but a bit clunky at showing the results and then in conjunction with Beyond Compare which is excellent at easily showing folder tree comparisons and manually adjust or deleting as needed etc. I verified everything in Finder to double check at the start as well. www.scootersoftware.com/features.phpBeyond Compare 30 day full function trial version, then $30 for the standard version which should be fine. apps.apple.com/gb/app/duplicate-file-fin...55628?mt=12Duplicate File Finder, find is free, some merge functions restricted to Pro version which is £202— I’m limited for disk space and can’t afford another £1,000 for 2 16TB drives, so hesitant to encode into ProRes or use Proxies. I’ll see how clunky it is, I’m am extremely slow editor so its not going slow me down much, as long as it doesn’t crash etc3— I’ll come back to the rest soon, I have another problem to solve first, I’ll put that in a separate post….