Welcome, Guest
Username: Password: Remember me

TOPIC: Multiple Libraries to one project. One library corrupted. Libraries contain references to each other

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 06 Sep 2019 06:03 #101293

Hi everyone. This is the first time I've posted here but I've been reading a lot and ya'll have saved me a lot of pain and suffering as I transition from 10 years of Premiere & Avid to FCPX. Love this program but I still have a lot to learn, especially media management.

I have been brought on to a documentary project that's being cut in FCPX. The project consists of 6 libraries, with tons of assets in each. Unfortunately, it seems, the editorial team has been moving Multicam sequences, compound clips, etc... into different libraries, copying the media from one library to the other. As a result, it seems all these libraries are referencing each other.

So I find myself working remotely in a less-than-ideal situation and the power goes out. Shuts down the footage drive, the separate SSD on which all of the libraries live and the Mac Pro I was editing on.

Once the power came back I was greeted with a warning that one of my libraries was corrupted. So I tried to open it from a backup but all **** has broken loose. Lots of missing files, spinning wheel of death, etc... on top of that, no matter what library I open they're all giving me the same warning (because they're all referencing each other)

I spoke to Apple support for two hours and all that got me was a stern talking-to about what I should have done instead.

So... my two questions are:

What can I do to fix this?
Once I fix it, what should I do to this entire project in order for this to never happen again? Do I need to consolidate all the libraries into one and make each library's contents its own event? Or is it OK to work with multiple libraries? And if so, what's the best way to work with multiple libraries?

Lastly, I apologize if this has been asked before. I tried to do my due diligence and study everything I could get my hands on but I couldn't find anything that addressed my particular situation.

Thanks,

Daniel
The administrator has disabled public write access.

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 08 Sep 2019 12:10 #101327

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1227
  • Thank you received: 256
  • Karma: 22
Daniel_Riser wrote:
... The project consists of 6 libraries, with tons of assets in each. Unfortunately, it seems, the editorial team has been moving Multicam sequences, compound clips, etc... into different libraries, copying the media from one library to the other. As a result, it seems all these libraries are referencing each other...the power goes out....I was greeted with a warning that one of my libraries was corrupted. So I tried to open it from a backup but all **** has broken loose. Lots of missing files, spinning wheel of death, etc... on top of that, no matter what library I open they're all giving me the same warning (because they're all referencing each other)...What can I do to fix this?
Once I fix it, what should I do to this entire project in order for this to never happen again? Do I need to consolidate all the libraries into one and make each library's contents its own event? Or is it OK to work with multiple libraries? And if so, what's the best way to work with multiple libraries?...

Daniel, sorry about the problems. This possibly started due to moving those items (bare) between libraries, then the power outage crash made it worse. It Unfortunately it's not officially documented but you should never do that, only move clips, projects, etc. *within* a transfer event. Not doing this can cause various issues, although the combination of that and an uncontrolled crash is unknown. Sam Mestman discusses this from 06:30 to 11:00 in the below video, but you may want to watch Sam's whole talk from 02:00 to 11:00. He demonstrates this on a Lumaforge NAS but it's not unique to a NAS.

hazu.io/pixelcorps/fcvug-7

Re how to fix it, that is difficult to say. Normally FCPX has excellent data integrity, even through many macOS or hardware crashes. The underlying SQLite database has transactional protection.

After any macOS crash or uncontrolled shutdown, run Disk Utility First Aid on all drives, including the boot drive. Running it on the boot drive should ideally be done in Recovery Mode (boot with CMD+R).

Then rebuild all Spotlight indexes on all drives: support.apple.com/en-us/HT201716

Neither of the above are expected to fix your problem, but they are standard procedures in a case like this.

Verify you have plenty of free disk space on all drives. An uncontrolled crash combined with a marginal out-of-space condition can cause problems.

If you have file-level backups (Time Machine, Carbon Copy, etc) of the library bundles, see if you can load those all as a single group. First save/rename your existing libraries.

Then try to load each library, one at a time. To avoid loading them all, launch FCPX while holding the OPT key, which gives you the option of what library to load.

If any single library won't load, state what the error message is. If it loads, delete all render and cache files for each library. Render files can be deleted via Files>Delete Generated Library Files>Delete All Render Files. Cache files cannot be deleted from within FCPX, only from opening the library bundle with Finder or by using the 3rd party tool Final Cut Library Manager: www.arcticwhiteness.com/finalcutlibrarymanager/

Anytime you experience anomalous FCPX behavior it may be good to reset FCPX preferences (see item 11): support.apple.com/en-us/HT203477

If all single libraries load one at a time, then try loading them all. If that works, make an immediate file-level backup of everything, or at least all the library bundles.

Then examine your clips in the Event Browser also your project and see if it looks OK. If you have red clips (missing media), examine that. If not but the project causes an error, try the undocumented fix command which is select the project, OPT+click on Clip>Verify and Repair Project.

The next step is copy all library events to separate events within a single library. I'd suggest creating a new destination library for this, and using the library inspector "Modify Settings" to define separate locations for media storage and cache.

If you are using proxies, I'd be tempted to delete those and regenerate them after the copy. That is just a gut feel. I know it can take lots of time. I have had problems previously when merging multiple libraries that use external proxies on different hard drives.

First name the events in each library appropriately to avoid name conflicts and confusion, e.g, name them libraryname_eventname. Then when they are all copied to one library you can tell where they came from. When doing the copy, answer "yes" to proxy/optimized media, whether this media exists or not. There is a data management bug involving audio files if you don't do this.

Please post how far you get in this procedure. Do not do anything you are not comfortable with.

How to avoid in the future:

- Have a UPS for your system and all hard drives.
- Ideally have all media for a given task on a single drive or array. This avoids the possibility of certain drives being off line during startup or restart, and also enables FCPX hard link optimization.
- Never move clips, projects, etc. between libraries unless they are within a transfer event.
- In general use a single library and multiple events for all related material. Yes this isn't always possible, especially if prior people have made multiple libraries.
- Use "lean libraries" and make multiple file-level backups. This means import all media using "leave files in place" and in the library inspector have Storage Locations>Modify Settings set to separate locations for media and cache. This keeps the library small, making it easy in Finder to right-click and duplicate it periodically.
- All media drives should be HFS+ or APFS. These are journaled file systems and more resilient to a crash.
- Run the latest versions of macOS and FCPX.
Last Edit: 08 Sep 2019 12:28 by joema.
The administrator has disabled public write access.

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 08 Sep 2019 15:06 #101330

  • Soran
  • Soran's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 87
  • Karma: 1
Thank you Joema for the detailed enlightening guide since I'd an issue in similar scenario in the past.

A couple of questions:

You say ' The next step is copy all library events to separate events within a single library. I'd suggest creating a new destination library for this, and using the library inspector "Modify Settings" to define separate locations for media storage and cache'.

1.How do you manage the bloated cash files once finished and archived the projects?

2.Do you use the same strategy, as in creating transfer Events between multiple Libraries if you need to go back to work with archived projects?

Regads, Soran.
The administrator has disabled public write access.

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 08 Sep 2019 16:09 #101331

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1227
  • Thank you received: 256
  • Karma: 22
Soran wrote:
...
1.How do you manage the bloated cash files once finished and archived the projects?

You don't need cache files when archiving or even before then. Cache files are very transient, short-term things. They will be automatically regenerated when needed. They can be safely deleted at any time. You especially want to delete them before archiving a project.

Soran wrote:
....Do you use the same strategy, as in creating transfer Events between multiple Libraries if you need to go back to work with archived projects?....

A good approach for archiving is create a new archive library configured for managed internal media, then drag/drop your project to a transfer event in the source library, then drag/drop that event to the archive library, then consolidate. Only the media needed for the project timeline will be copied to the archive library. This eliminates the concern over later loading the archive library and not having all media for the timeline. Whatever media in whatever events that project references (and only that media) will be copied to the archive library.

If you later restore that archived library and want to work with other libraries, you should only move clips and projects between libraries if inside a transfer event.
The administrator has disabled public write access.

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 08 Sep 2019 18:28 #101333

  • Soran
  • Soran's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 87
  • Karma: 1
Extremely useful tips! Many thanks!
Here's my scenario Joema,
I've x5 Episodes, all archived and all media in Library.
Now I want to create a new 6th Episode with the new content, however, I also need to pull up some elements from previous episodes that were archived separately. Those elements are Compound Clips, parts of the projects and other media etc.

I guess, I need to create a transfer Event for each previous archived Episode/Library to copy the media from the older ones to the new one that has also it's media in Library?
The administrator has disabled public write access.

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 08 Sep 2019 19:32 #101334

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1227
  • Thank you received: 256
  • Karma: 22
Soran wrote:
...I guess, I need to create a transfer Event for each previous archived Episode/Library to copy the media from the older ones to the new one that has also it's media in Library?

If you now or in the future envision needing material from previous episodes, those should ideally be part of a single library. E.g, you could have one season of episodes in a single library, each in a separate event. Having them within one library (1) Potentially saves space since any project in any event within a library can reference all material in the library without copying clips - although FCPX has optimizations to avoid this in several cases (2) Enables library-wide querying. You cannot query for material across several libraries.

Since you already have a library per episode it might be better to merge that to a single library as previously mentioned. Yes one way is creating a transfer event within each season's library, copying (not moving) the required clips, projects, etc to that transfer event which is then copied to the combined library. When you do this be careful you are copying not moving the events.

When you copy clips to another event within a managed library it does not actually duplicate the media. Rather it creates another CurrentVersion.fcpevent database file which points to the original media within the library. It will even do this when copying the transfer event to the combined library. This is analogous to doing so with external media where it creates additional symlinks. In addition FCPX will where possible also use HFS+ hard links to minimize media duplication (if within the same volume).

You don't need to remember all the rules and behaviors, only that FCPX will wherever possible provide the appearance of a duplicated file but without the actual space consumption. Again, this is within the same disk volume so this is another reason it makes sense where possible to have all media on a single large drive or array.

While having all media within the library (aka a managed library) has some benefits (esp for archiving), for normal use may can complicate things. The library bundle is very large since it contains media, cache and maybe proxies and optimized media. It is almost impossible to duplicate the huge library for a file-level backup. It also slows down ingest since the media must be physically copied to within the library vs import with "leave files in place". It is harder for multiple libraries to share the same set of media. That said, some media types such as AVCHD should not be imported using "leave files in place".

Another approach is have a standardized folder structure for your offloaded media, then import to FCPX using "leave files in place". For media like AVCHD that requires re-wrapping, re-wrap this externally using the 3rd party tool EditReady then import to FCPX using "leave files in place". Then create the library and within FCPX define specific external locations for storage and cache. If you import using "leave files in place", that storage is only used for generated proxies/optimized media. The library bundle remains very small and in Finder you can easily right-click and select "duplicate" for a backup. I do this anytime I load an XML file or do any FCPX action I need an easy rollback for. You can do this whenever important work is committed, then copy the small library file to external storage of the cloud for backup. You don't need to do this now, it's just something to think about for the future.
Last Edit: 08 Sep 2019 19:38 by joema.
The administrator has disabled public write access.

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 09 Sep 2019 05:15 #101338

Wow. I can’t begin to put into words how thankful I am for that incredible response. Your advice saved the day. The multi-library nightmare has been consolidated and everything was restored and is now running beautifully. I also bought the biggest, best UPS I could afford. #neveragain

The only thing I can’t seem to find Information on are transfer events. I’m assuming a transfer event is just a new event you have designated only for the project(s) you want to transfer, or are they a special type of event? I tried to find information on that but couldn’t find anything documented.

Thanks again. I’m so grateful.
The administrator has disabled public write access.

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 10 Sep 2019 11:00 #101357

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1227
  • Thank you received: 256
  • Karma: 22
Daniel_Riser wrote:
...The only thing I can’t seem to find Information on are transfer events. I’m assuming a transfer event is just a new event you have designated only for the project(s) you want to transfer, or are they a special type of event? I tried to find information on that but couldn’t find anything documented....

Yes, a "transfer event" is just a made-up name for an event used to transfer data between libraries. This is discussed from 06:30 to 11:00 in the below video:

hazu.io/pixelcorps/fcvug-7
Last Edit: 10 Sep 2019 11:01 by joema.
The administrator has disabled public write access.

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 11 Sep 2019 20:49 #101389

  • Soran
  • Soran's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 87
  • Karma: 1
Yes, soconded thanks for this thread and the insightful answer Joema that's very through...

Although it deserves a new thread, I can't help asking since it'll be relevant to the topic being discussed...

If I create a new non-managed library , keeping files in place, that'll be used over and over as an improvised MAM or DAM, does the library updates itself sync whenever I add media to files in place? Similar to sync function of watch folder behaviour in DAM's...

Much appreciated for the tips..
The administrator has disabled public write access.

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 11 Sep 2019 23:16 #101390

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1227
  • Thank you received: 256
  • Karma: 22
Soran wrote:
...If I create a new non-managed library , keeping files in place, that'll be used over and over as an improvised MAM or DAM, does the library updates itself sync whenever I add media to files in place? Similar to sync function of watch folder behaviour in DAM's....

An FCPX library only knows about material that has been imported. During import, numerous characteristics are recorded in SQL database tables about each media file. If you change those files it will produce an error. If you add additional files using Finder to the media folder, FCPX will not know about those unless they are imported.

If you want an improvised MAM, one way is keyword the material then use FindrCat to embed this as Finder tags. This enables searching those using Finder search tools including boolean queries. In theory you could even use NeoFinder to search for those Finder tags across multiple off-line hard drives: www.cdfinder.de/guide/8/8.7/neofinder_tags.html

However using KeyFLow Pro 2 is probably a more integrated method: www.keyflowpro.com

Searching multiple libraries using KeyFlow Pro 2:
The administrator has disabled public write access.

Multiple Libraries to one project. One library corrupted. Libraries contain references to each other 15 Sep 2019 21:58 #101452

  • Soran
  • Soran's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 87
  • Karma: 1
KeyFlow pro; it sounds like a lifesaver! Definitely, I willl seriously consider especially due to It's multi-library search features... I wish FCPX has that feature in coming upgrade.
But how does it deal with the metadata:
1. imported CCs buried within libraries from FCPX?

2. Imported from Apple Photos, other DAMs or previously tagged sources?

Once again, many thanks for the support..
Last Edit: 15 Sep 2019 22:01 by Soran.
The administrator has disabled public write access.