TOPIC:

Hi all,I'm wondering if anyone can help - I'm about to go into the final week of a pretty long edit and FCP has started behaving really badly. Moreless anytime I click on an event within the library it stalls, showing a loading screen and also freezing with the spinning beach ball.I've tried copying the library to a Lacie 5B drive, which has not been of any help - I fear that the library may have become corrupted during an ingest process/a plug in is causing it to behave like that (I have the dehancer grading software installed, but am not using it for my main edit as it's so processing heavy).I'm working off my built in 4TB SSD. Below is my macbook spec and have also attached a recent crash report. If anyone has any suggestions I would be really grateful!

James

Spec:
MacBook Pro 16-inch, 2021
Chip Apple M1 Max
Memory 64 GB
macOS Ventura 13.3.1 (a)

FCPX 10.6.5

James, thanks for providing the crash log.

It may be that there is an issue with a font. Use Font Book's Validate Fonts function (in the File menu) to check your fonts. There could be a corrupt font cache, as well, which you can resolve by starting up your Mac in Safe Mode, logging into your account, then restarting normally:

Start up your Mac in safe mode - Apple Support

jamesh001 wrote:

...I have the dehancer grading software installed, but am not using it for my main edit as it's so processing heavy...

I'd suggest following Dave's advice about checking the fonts.

Besides that, the Dehancer website says "Compatible with Apple Silicon (using Rosetta)". That may mean it's NOT compatible with Apple Silicon unless you force the Apple Silicon version of FCP to run in Rosetta mode. Regardless of whether you are using Dehancer for the main edit, are you running FCP in that mode?

The crash dump shows there are several Rosetta processes in memory, including DehancerProFCP_XPC. I'm not sure of the relevance of that.

BTW this crash dump is really a type of spin dump, so technically nothing crashed, but FCP was hung. The "heaviest" thread is apparently the main UI thread, which also runs the UI. That is why it hung with a beachball -- the thread was hung waiting on something and could not update the UI.

That thread was apparently trying to load some project data and became hung waiting on something.

The other interesting thread is thread 0x100db (the long one). Early in that backtrace are references to DSStore which is the FCP private framework that wraps the Core Data SQLite persistent data store calls. It then goes to NSSQLCore calls, then NSSQLite calls, then sqslite3 calls, then sqlite3 B*tree calls (typically used for SQL indexes), then unixRead (an I/O call), then lck_mtx_sleep (implying lock mutex sleep), then finally kernel.release.t6000.

The lck_mtx_sleep call implies that the thread is waiting for a lock to be released. This is normal for I/O operations, especially for database reads and writes, as they need to ensure data consistency and prevent data corruption.
Seeing this trace doesn't directly imply database damage. It just tells us the operation was waiting for something when the stack trace was collected. This could be due to a normal delay in reading from disk, or it could be due to the database being locked for a write operation, or due to contention for resources with other threads or processes.

In general I'd suggest upgrading to MacOS 13.4. While you normally don't want to upgrade in the middle of a project, that is mainly valid to avoid problems. You are already facing a severe problem. I did some preliminary tests with FCP 10.6.5 on Ventura 13.4 and performance seems good. In some cases on complex timelines it seems better than 13.3.1(a), but that is only results from one machine, a M1 Ultra Mac Studio. If your problem is caused by Dehancer or some other plugin, that might happen regardless of what MacOS version.

Thanks so much for this! Unfortunately I've tried all of these methods with no luck. It's also worth noting that I've deleted some unused media and I think it started to slow down after I did this...I did it via FCP to clear some HDD space, but then I think the library began to slow down and hang after this.

Have you guys ever heard of FCP doing this before? Do you think there is a solution?

Your help is really appreciated

jamesh001 wrote:

...Have you guys ever heard of FCP doing this before? Do you think there is a solution?..

In your case "doing this" is a broad description of a beachball when you do certain things like click on an event. There is no one problem that causes that kind of behavior. It could be a plugin, or damaged data, or a damaged project or a slow hard disk or an inefficiency in FCP or the underlying MacOS that manifests under a complex set of conditions.

Just taking a guess, if it happens when clicking on an event, that might imply you have many projects or snapshots. There is no simple limit but if you have a large number of those, esp. if combined with other problems it can make the system slow. Each project is stored as a separate SQLite database and FCP apparently tries to transparently pre-open some of those in certain cases. That takes overhead and time, esp. if the projects are complex.

When you say it formerly ran OK but it's a "pretty long edit", how long and how complex is that? How many clips in the project?

FCP can slow down considerably if the timeline is very complex and over about 1 hr long. However I've seen apparent improvements of FCP 10.6.5 on Ventura 13.4, but that version has a major problem with 100 fps and 120 fps Sony 10-bit 4:2:2 XAVC-S. But except for that and the already-existing AU Validation problem, FCP 10.6.5 seems to run better on Ventura 13.4, especially for complex timelines.

There's over 30 hours worth of 4k XF-AVC 422 Long Gop footage.
I've used a bunch of Multicam

Here's a link to the library: www.dropbox.com/s/hsgm99i2sw7lcvy/202305...946_GMT%2B1.zip?dl=0

It seems to be really misbehaving even when it has no media connected! If anyone could offer any solutions or advice I would be ever so grateful. I'm really not able to get my head around the problem.

James, please provide more details on the history of this problem. Did it formerly work mostly OK? If so, when? Was the onset of poor performance sudden? Did it happen immediately after some specific system config change, data change, new data import, etc?

Or was it a gradual slowdown which progressively got worse over a period of many days or weeks?

The problem is caused by the "Cuts" event. Most of those projects do not appear excessively long or complex from a timeline standpoint, but there is something wrong inside the SQLite databases which comprise each project. E.g, it's typical that the ZCOLLECTIONS table inside that database might have a few hundred thousand rows for a very large project. Yours have 5 *million* rows in each one, and most of rows appear to be anomalously duplicated. I have never seen that before.

The SQLite database for one such project was not technically corrupt -- it passed the SQLite PRAGMA integrity_check command. But the number and contents of those rows is highly abnormal.

Not only is FCP slow, but every software tool that touches that library is slow. E.g, I can use a SQLite tool to export the data as .csv -- that takes a long time. I can't even open it with Excel because it has over 5M rows and Excel only supports 1M rows. It's actually a miracle that FCP even runs in this situation.

It took a long time but I exported one of those project XMLs. Below is an excerpt of the anomalous XML. I only include a portion, but there are over 100,000 repetitive lines referencing the clip name "A018C114_230317FE_CANON". I have no idea how this could happen.

I don't know what originally caused it but the slow behavior is not a FCP bug, system config issue, slow disk, etc. If you loaded those project XMLs in Resolve it would probably behave similarly or even crash.

Obviously the question is how to fix it. If you deleted the Cuts event, that might fix it, but I don't know from a data standpoint if that's possible. If you have older Time Machine backups or FCP auto-backups (which by default are in /Movies/Final Cut Backups, it might be possible to load versions of those before this problem happened.

See attached .txt file containing XML excerpt from project BBCSW_AMR_POST CLIENT CUT 1 Snapshot 16 May 2023 at 11/06/43

Joe, some comments from what I've noticed with the Library that James posted...

- It took a looong time for the Library to load. FCP is using over 35 GB of RAM.

- I looked at "BBCSW_AMR_POST CLIENT CUT 7" in particular (it showed as being the most recently modified Project in the Browser in the "! Cuts" Event). I ran the "secret" Verify and Repair Project command and that took 20+ minutes to fix the "inconsistencies" that were found.

- I exported an XML for the above Project/timeline and it took maybe 30 seconds to export. The XML file weighs in at 161, 250 lines (in BBEdit) with 19,219 matches for "A018C114_230317FE_CANON".

- The seemingly repetitive lines are actually unique, it seems, with the "offset" value incrementing by 6/25s in each instance. The last offset value I see is "11519400/2500s" which is about 77 min. If that is longer than the actual duration of the file/clip, then that might be relevant. For clarity, this is the exported XML after I "verifed and repaired" the Project/timeline first. Note: it seems from the FCP Inspector that the media duration for the associated file is 1h16m48s.

- The potential problem clip/file is "A018C114_230317FE_CANON.mov". The file/clip was renamed in FCP to "Jon Stokes - Clean" and "Jon Stokes - Clean 1", if not others that I haven't noticed yet.

- The "asset clip" item repeated in the XML file seems to refer to the same metadata tag for "Reel", in that (as mentioned above) the offset value is the only difference.

---

It's difficult to examine things as each Event I click on is taking many minutes to load. I've had to force quit FCP a few times.

Hope this helps...

Dave and Joe -

Thank you so much for this - you're both absolute legends!

I'm currently in the process of deleting all of the clips from the compounds 'Jon Stokes - Clean". (The last time I mask out a boom mic for for a complete interview...)

I think I've managed to get rid of most of them (as I've deleted every project other than 'BBCSW_AMR_POST CLIENT CUT 7', but it's still pretty sluggish. Would I be right in thinking there will be duplicates of these compound clips for each snapshot I have created and this could still be causing the really slow loading times, even if the projects have been deleted?

Dave: Thanks; I overlooked the "offset" value incrementing by 6/25s in each instance. What you did gives me some ideas to pursue tomorrow. E.g, for the "BBCSW_AMR_POST CLIENT CUT 7" project's CurrentVersion.fcpevent file (which contains the SQLite database) I will export the .csv of the ZCOLLECTION table. Then I'll run the repair option, then re-export that table and use Beyond Compare to diff those and see what it fixed.

The utility I use to examine the SQLite databases is SQLPro for SQLite. Unfortunately, most of the tables within CurrentVersion.fcpevent are difficult to interpret, and one contains binary data, but ZCOLLECTION has some vaguely understandable items.: apps.apple.com/us/app/sqlpro-for-sqlite/id586001240?mt=12

Yes, with BBEdit I don't think I could go through those gigantic XML files.

Agreed, it takes a long time to load. Now I know why: the tables are gigantic and vastly disproportionate to the timeline size and complexity. This examination is actually a useful endeavor.

James, that's a good point. Unlike project duplicates, snapshots freeze the state of all compound and multicam clips in a project. Obviously, that state information must be stored somewhere. That includes every edit and every effect on every clip. But for compound & multicam clips, a snapshot severs the inheritance to the parent clips, so that likely entails more state information stored (in the SQL database). It is possible there's an exponential performance cost if making snapshots of complex timelines which contain lots of compound and/or multicam clips.

There is no question the stack traces show the FCP thread accessing the SQLite database is bogged down. I did many different spindumps during beachball periods when opening events in your library. The SQLite database thread was bogged down in each case, and the UI thread was apparently waiting on it, thus causing the beachball.

I have that library on the internal SSD of my M1 Ultra Mac Studio. Even then it is very slow to open events or do searches.

Merely examining the number of projects, and length and superficial complexity of the timelines would not reveal the problem cause in an obvious way. In this case a tip-off is the slow opening.

I have never internally examined the before & after SQL state following a snapshot, but I'll do that tomorrow and try to quantify the parameters and boundaries.

James, if the other projects *and* snapshots are deleted, it's possible it might get faster. However I'm not sure what cleanup and garbage collection algorithms are involved for the SQL tables. I will also examine that tomorrow.

(Joe already responded before I finished this post.)

Hey James, that's a good thought about the compound clips. If some compound clips, or their component clips, are located in other Events, those Events may need to load even when you are just working in the "! Cuts" Event. I can't say for sure since your Library is giving me some issues (which is hindering my attempts at trying to troubleshoot things)...

I still wonder about that "A018C114_230317FE_CANON.mov" file and the clips that reference it (and why the seemingly spurious "Reel" metadata tag is repeated in the XML file — you'd think that the Reel metadata would only be one instance for the file).

If you're actually still editing, you could always turn off various effects in the Inspector. Then, turn them back on for finishing. That way, FCP should be more responsive (generally speaking, of course) when you need to do editing of clips in the timeline.

Once you've cleaned up things a bit, you could try copying the Event with the Project/timeline into a new Library and see if that helps.

You might also use the "secret" Project fix command (hold option and select the Clip menu and choose "Verify and Repair Project").

Joe will most likely chime in with some other (and better) info to help further...

joema wrote:

The utility I use to examine the SQLite databases is SQLPro for SQLite. Unfortunately, most of the tables within CurrentVersion.fcpevent are difficult to interpret, and one contains binary data, but ZCOLLECTION has some vaguely understandable items.: apps.apple.com/us/app/sqlpro-for-sqlite/id586001240?mt=12

Yes, with BBEdit I don't think I could go through those gigantic XML files.

I don't need to be tempted with an SQL application... I'll leave that stuff to you. But, thanks for mentioning it.

I've opened some pretty huge files in BBEdit. I can't recall but I believe there is another text editor I've used that could open absolutely huge files...

Look forward to at least reading more about this issue... Thanks, Joe.

Hey, Joe, in lieu of an offline discussion (and somewhat related to complex Lbraries as James has), I wonder if you have any thoughts about using "Library versioning" instead of Snapshots and copies of Projects/timelines within an FCP Library?

By this, I mean only making a small number of Snapshots or copies of Projects while working in FCP, then periodically, with FCP quit, make Finder copies of Library bundles. Of course, this would assume that media is stored outside the Library. This way, you'd end up with five, ten, or twenty odd copies of the Library itself, rather than lots of "versions" of things inside the Library.

I wonder if this would work well to keep the Library "slim". Add your other advice about minimizing Events and making good use of keywords/metadata for organization, etc. I guess one might refer to this as preferring smart collections to Events when organizing in FCP.

Any thoughts? Thanks.

Thanks against guys. I think you may have solves the slow library problem...but when one problem is solved another arises!

All of my multicam clips in the project have now greyed out and are no longer playable/linkable. When I try and 'Reveal in Browser' a message pops up saying ' Failed to reveal clip' 'The original clip could not be found or reconstructed'.

Do either of you have any ideas what this may be and how I would get them back?

I've attached screen shots of the timeline and error message.

jamesh001 wrote:

Thanks against guys. I think you may have solves the slow library problem...but when one problem is solved another arises!

All of my multicam clips in the project have now greyed out and are no longer playable/linkable. When I try and 'Reveal in Browser' a message pops up saying ' Failed to reveal clip' 'The original clip could not be found or reconstructed'.

Do either of you have any ideas what this may be and how I would get them back?

I've attached screen shots of the timeline and error message.

The only thing that comes to mind (sorry) is that you might want to check that your "Dialogue" or other roles used in the MC clip are active/turned on (via the Timeline Index). Also, you might want to check to make sure solo-ing isn't active and inspect the clips inside of the MC clip you're using (for disabled clips). Check things in the angle editor...

James and Joe, I had a few minutes to spare and tried a "quick fix" to the XML file I exported for the "CUT 7" Project. I removed the repetitive lines that Joe first discovered.

I still wonder if the "Long GOP" nature of the source files along with a configuration of the Library is triggering some sort of edge case that causes spurious "recursive" calls in the underlying database where those repetitive lines in the XML are a manifestation.

I imported that XML back into FCP (in a different Library). The Project loaded in about 20 seconds. The Library that James posted was taking almost 30 minutes to load on my Mac Pro. That "CUT 7" Project also would take around 20 minutes to process the "Verify and Repair" command. The full Library also took around 30 minutes to close (leaving FCP running afterwards with no Libraries opened).

Without media, missing LUTs, etc. (of course), I was able to move around the Project/timeline quite well. There were no lags or "beachballs". The FCP UI was very responsive and fluid. Searches were very quick.

---

Back to the multicam clip issue...

I recall that Tangier Clarke had a similar issue with multicam clips (MCs) turning grey (and some clips inside of the MCs). I couldn't find any mention of whether he had found a solution, or not.

So after spending almost 4 hours deleting all of the rogue compound clips and suspect events, the library is running as smooth as ever.

I managed to get those greyed out multicam clips back by opening an old back up and they came back. It seems as if they somehow lost all of their links.

Dave and Joe, thanks again for all of your support with this. I really was lost and would never have considered the compound clip had you not have addressed the clip that was looped within it.

If there’s a way of private messaging on this - I’d love to send you a gift as this has saved me from constructing a whole edit again!!

DaveM wrote:

... triggering some sort of edge case that causes spurious "recursive" calls in the underlying database where those repetitive lines in the XML are a manifestation...

Good question. The XML is reflecting what is in the database. There are several tables which are relationally linked by primary key <-> foreign key relationships. It's not just the 'ZCOLLECTION' table that has excessive rows (5 M), but 'ZCOLLECTIONMD" (MD=MetaData) has 1.4 M rows, and 'Z_3CHILDCOLLECTIONS' has 6.6 M rows.

Several of the columns in those tables are indexed, so each update forces multiple additional index writes. Here is a link to an annotated extract I did: joema.smugmug.com/FCP-Hang-Metadata-Study/n-F53QqF/

Since the schema is not documented and I don't have the data dictionary, deciphering raw data from the tables is limited. The goal is not just fix this specific case but understand how it happened.

It will be important to determine did James do something "procedurally normal" that produced a technically correct data state but caused extreme performance degradation? Or did he do something normal that triggered an edge case where the database and/or FCP went haywire and added a gigantic number of spurious rows to the SQL tables?

We've already made a lot of progress. There is no question the immediate cause of the performance problem is the data state of the SQL tables. The XML of that project definitely looks abnormal.

DaveM wrote:

...I recall that Tangier Clarke had a similar issue with multicam clips (MCs) turning grey (and some clips inside of the MCs). I couldn't find any mention of whether he had found a solution, or not.

I don't remember that specific symptom. He's had multiple performance issues which I've worked on a lot. I currently believe the source of his issues are different from James' case. XCode Instruments traces I took while running his library implied it was a slowdown related to FCP single-thread processing of complex timeline metadata. However at that time I didn't have the understanding of this new "huge XML / huge SQL table" problem.

I'm about 70% certain the two problems have different sources, but I need to go back and check Tangier's library again.

jamesh001 wrote:

I managed to get those greyed out multicam clips back by opening an old back up and they came back. It seems as if they somehow lost all of their links.

I wish I knew more details about this, and how it got fixed. I've never run into it personally (I seem to avoid most issues by almost always exclusively working with media either optimized in FCP, or transcoded to ProRes prior to use in FCP). I'll just have to keep a mental note of it, as I'm about to begin a feature-length project.

I'm glad some of what we discussed helped you to sort things. You're welcome. There is a way to send a PM or an e-mail from your profile page, but with my OS set to dark mode, there is no icon/widget visible to click on. I noticed the cursor change when hovering over the spot:



I have noticed that some people have had problems with PMs or e-mail actually working. Some of us have contact info in our profiles. The thoughtfulness is appreciated, in any event...

Cheers.

jamesh001 wrote:

So after spending almost 4 hours deleting all of the rogue compound clips and suspect events, the library is running as smooth as ever.

I managed to get those greyed out multicam clips back by opening an old back up and they came back. It seems as if they somehow lost all of their links...

James, I'm very glad this had a good outcome. In my last post I was doing the groundwork for a much longer troubleshooting process. It would help if you could please give us some more details on your corrective steps.

When you say "deleted all of the rogue compound clips", in what sense were they rogue? How did they get there? How did you know which ones to delete? With everything you now know, do you have any idea (from a procedural standpoint) how the problem happened?

Ditto for the "suspect events". I know the Cuts event was "suspect" in that it had those projects with gigantic CurrentVersion.fcpevent databases. How did you decide what events to delete?