TOPIC:

Hello!

I hope to get some information to solve the issue I am having since this a show stopper using FCP in the future.
AS of 2023 Synology Drive was implemented to to sync all our Projects, Libraries and Media Files from our Servers to our local Raids.

The problem I am to we are having is that, when a FCP Library is being synced or even stored in the sync location.
Media is shown offline, modified to with the missing plugin warning. If I move this Library out of the sync location relink the media this bring no issues.

It seems FCP Library and the Synology Drive syncs is causing issues with the FCP Library files (Database)?

Has anyone als has this issue or best a fix so I can pass this on.
As many business work across the board no other user to software reproduces this matter, only FCP.
So its ether a fix or Adobe Premiere!

Thank yo

Hi,

we are successfully using the synology add-on "USB Copy" to do nightly backups to our attached external harddrives. The problems you describe are not easy to solve and some testing is required. I remember, for example, that we had an issue with special characters in snapshot filenames that didn't translate to unix conventions. I sincerely hope you can solve your problems before contemplating a switch.

AppleGuru wrote:

...a FCP Library is being synced or even stored in the sync location...Media is shown offline, modified to with the missing plugin warning. If I move this Library out of the sync location relink the media this bring no issues....As many business work across the board no other user to software reproduces this matter, only FCP. So its ether a fix or Adobe Premiere!...

One possibility involves some logical data damage caused by another task changing the FCP library internal structure while FCP is trying to use it. Unlike Windows, most UNIX-heritage file systems do not have mandatory file locking, only advisory locking. That means if you back up or copy a transactional SQL database file such as the FCP library while FCP on up and in use, it may end up in an inconsistent state. If the Synology software is copying or syncing and FCP library while FCP is up and on line, that could happen. Even if no user is accessing the NAS-hosted FCP library, you must always assume that FCP could be doing I/O of some type.

If you mean the FCP library was copied to or from the NAS while FCP was down, the integrity of that copy should be verified. The library is not a single file but a complex bundle containing many data objects. We use Hedge or some other data offloading tool when copying critical camera data, and those are just simple flat files. When copying far more complex data objects such as a FCP library, the copy should ideally be validated or checksummed. There are various methods and tools for that.

Another possibility is the library is actually OK and the NAS is not supporting all the needed APIs requested by FCP. That can differ from one app to another, so what a particular app such as Premiere or Resolve does is not a reliable guide.

For example say FCP makes an I/O call concerning a Finder tag. That call is redirected to the network layer. The NAS may not fully support Finder tags hence causes an error. Maybe Premiere or Resolve don't use Finder tags and they get no error. It is not incumbent on FCP to quit using Finder tags, rather it's the responsibility of the NAS to support the complete spectrum of all I/O requests that could occur in a modern Mac environment. Finder tags are just an easy to understand example. The same situation could arise regarding multithreadded and overlapped I/O requests, depth of disk queue, how certain return codes are handled for I/O calls, file permissions, etc.

I have never seen a similar problem report from customers using FCP libraries on a Lumaforge NAS server. Therefore I don't think there is a generic problem with FCP on a NAS.

First of all thank you for the replies!

Unfortunately I don't understand what to do to fix my issue.
So we are using the Synology Drive Client app to sync pur locals RAID with the Servers.
Today I gave it another try and receive the error that a project was damaged or couldn't not read it.

I also removed the rule not to sync .tmp files

Perhaps not the best or most targeted advice, but I did a search and found several sources of info about using Synology with FCP:

Using Synology NAS with Final Cut Pro

Thank you, unfortunately they do not cover Synology Drive.
But I did some testing at home at it seems to work fine. My guess is that it might be the Synology disk format.
I have ext4 at home and in the studio I know for a fact they are using Btrfs.

This kinda sucks but I guess, we might have to move over to another NLE 😕

AppleGuru wrote:

Thank you, unfortunately they do not cover Synology Drive.
But I did some testing at home at it seems to work fine. My guess is that it might be the Synology disk format.
I have ext4 at home and in the studio I know for a fact they are using Btrfs.

This kinda sucks but I guess, we might have to move over to another NLE 😕

I don't think ext4 or Btrfs is related to the problem. Those are server-side filesystems used by a Synology NAS.

This is likely an operational problem introduced when you began using Synology Drive to sync complex open FCP libraries while in use. An FCP library is internally a relational SQL database consisting of many files. Therefore, it should be considered transactionally inconsistent unless FCP is shut down.

The situation is similar to having a Windows client running SQL Server and trying to sync that SQL database to a Synology server while writes to the SQL transaction log are happening. In either this case or the FCP case, it is not a bug with the app, and there is no fix possible in the app. Instead, it is a procedural issue that requires understanding the nature of transactional consistency during backup procedures and taking appropriate action.

Test results will naturally vary based on the activity level in the FCP library. For example, you could run one test where Synology Drive does not corrupt data, yet in another test, it might. But the only reliable method is shutting down FCP before the library is backed up, transferred, or copied.

Due to the complex internal structure of the library, it's also wise to consider doing a verified copy using any of the various utilities such as Beyond Compare: www.scootersoftware.com/ Hedge: hedge.video/hedge , Carbon Copy: bombich.com/ or make a validation checksum using the tool Integrity Checker: diglloydtools.com/integritychecker.html

Other NLEs (such as Premiere) that are not based on a SQL database and have limited data organizational features might temporarily avoid having their project database damaged by Synology Drive. However, it could later happen. I suspect damage would be more likely with Resolve since it also uses SQLite but not to the degree of FCP. Resolve can optionally use PostgreSQL to store project data. If Synology Drive tried to sync that while in use, it would possibly damage the database similar to this FCP scenario.

Your best steps are to verify that no FCP library is backed up or synced while it is open and in use. When backing up or syncing a library through the network, it should ideally be verified or checksummed. If you only want a library or project moved to another machine, that can be done by exporting and importing XMLs.

Okay I understand, but this error has happen not having FCP open.
The Library worked fine and then copied to the Server then synced to other Edit Station , when opening the error is displaced. So my guess was the there was a error during the Sync "copy" process.

AppleGuru wrote:

Okay I understand, but this error has happen not having FCP open.
The Library worked fine and then copied to the Server then synced to other Edit Station , when opening the error is displaced. So my guess was the there was a error during the Sync "copy" process.

You said "If I move this Library out of the sync location relink the media this bring no issues".

Does that mean you move the library to another location on the Synology NAS, or you move it back to a local drive attached to the Mac client?

You previously said when the library is moved to the sync location (I assume that is a certain folder on the NAS), FCP has an error about "Media is shown offline, modified". The "modified" error means from FCP's standpoint the file has been altered. Synology drive might have done someting seemingly innocuous such as changing some flag in the media file header, but from FCP's standpoint the file has changed. From a safety standpoint it will not open the file.

It's likely if at that moment you used a binary file comparison tool between the FCP library on the sync location vs a copy of the FCP library on a local hard drive, it would show that Synology Drive changed something inside the library. On such tool is Beyond Compare which can compare many types of files including binary files: www.scootersoftware.com/

Anytime you deal with a NAS, there are also possible issues with SMB vs NFS, which works best with certain apps, various NAS tuning parmeters, etc. It's also possible that might be involved. None of that affects FCP except as the victim of those changes. There is no change to FCP source code that can fix this since the problem was caused by Synology.

Sorry for the confusion.

Let me try to expel again.
If I take FCP Library and open it directly from the Synology over SMB there no issues.
But if I sync it to my Mac using the Drive Client the Library and op this in Final Cut Pro X it will prompt errors.
Ether a project damaged or issues with linked media.

For example with the linked media it will prompt modified files.
Now when relink them the other media that is linked become unlinked with the same error message.
I added two screenshots...


The error happen as soon the libraries are either synced down to the Mac or copied in the location of the Sync folder.

P.S. I also noticed when I pause the Drive Client and then open up the FCPX library I can relink the media with no problem. As soon as I resume the client I get error again modified media.

AppleGuru wrote:

...If I take FCP Library and open it directly from the Synology over SMB there no issues.
But if I sync it to my Mac using the Drive Client the Library and op this in Final Cut Pro X it will prompt errors.
Ether a project damaged or issues with linked media...

On Windows if you open a file, another application cannot open it for writing. The app does not have to issue any API call -- it's built in to the file system, called "mandatory locking". By contrast MacOS and other Unix-heritage applications use "advisory locking". There is no lock-based enforcement preventing multiple apps from opening the same file, even for writing.

If Synology Drive opens the FCP library (which is internally a SQL database), it can interfere with I/O from FCP. Or Synology Drive can write to that database file. Even something as simple as a temporary .tmp file could cause problems.

This is discussed in Synology's FAQ where it says "...Avoid syncing file types that are constantly being accessed. Some files are constantly being accessed due to their nature, such as log files, virtual disks, and databases. We do not recommend syncing such files since syncing them via Synology Drive Client is likely to fail."kb.synology.com/en-us/DSM/tutorial/What_...syncing_Drive_Client

This is not something that can be fixed in FCP. Your network people caused this problem, and they need to fix it. The above article says if you have problems in this area "please contact Synology Technical Support".

There are numerous methods your network people can use to troubleshoot this. They can use Wireshark or similar protocol analyzers. They can use Synology's "File Auditing" feature to determine which Synology process is accessing the FCP library files while FCP is using those.

If the library is apparently damaged, you can use Beyond Compare to determine which file within the library was changed or damaged vs the initial state. You'd need an initial file-level copy of the library to compare against. www.scootersoftware.com/

Thank you for your input!
We talked to the IT guys and we decided not to sync FCP Libraries. I wonder if we can find away to block the library from syncing. It seems the Synology Drive does not see the bundle as a file.

Headaches to come