Welcome, Guest
Username: Password: Remember me

TOPIC: Spread media across hard drives

Spread media across hard drives 21 Dec 2017 13:02 #92641

  • Xavier Novembre
  • Xavier Novembre's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 42
  • Thank you received: 1
  • Karma: 1
did you observe that spreading media across hard drives would slow down the app ?

as well as keeping a light library on internal SSD et everything else on an fast external volume
The administrator has disabled public write access.

Spread media across hard drives 21 Dec 2017 13:10 #92642

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2944
  • Thank you received: 391
  • Karma: 34
Spreading media accessed by a single Library does slow it down. They system isn't accessing one drive, but several. that's several different drive calls, each with it's own file request. Rather than all file requests on on drive. The except is in a RAID configuration.

The internal SSD drive is theoretically the fastest on your system. BUT, if you're working with larger Libraries and Project Timelines, you can easily eat up the bandwidth of that drive. Every drive can only read/write so much data per second. Just like your Internet access bandwidth. The operating system and applications write/read/delete invisible working files non-stop, very fast, and they need some of that bandwidth to do that quick enough to give you top performance. A big HD or 4K project can eat up that bandwidth enough so that all that traffic slows down your system drive.

This is why it is recommended to place all media and Libraries on an external, very fast secondary drive. If you use multiple drives, place media for each Library on the same drive as the Library. We've tested this over and over at a TV station I used to work with, and it holds up as true. Others may have different results.

Multiple drives isn't bad, only when you're Library is on one, and the media it must access is spread across 3 or more. Especially if those drives are slow, or connected via slow connections (USB 1, Firewire).
The administrator has disabled public write access.

Spread media across hard drives 21 Dec 2017 17:52 #92645

I put my media library and cache on two separate drives..
Good idea or nay?
The administrator has disabled public write access.

Spread media across hard drives 21 Dec 2017 19:35 #92647

  • Xavier Novembre
  • Xavier Novembre's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 42
  • Thank you received: 1
  • Karma: 1
NotARubicon_Productions wrote:
I put my media library and cache on two separate drives..
Good idea or nay?

doesn't sound like a good one...
The administrator has disabled public write access.

Spread media across hard drives 22 Dec 2017 01:23 #92651

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2944
  • Thank you received: 391
  • Karma: 34
If they're both fast enough, it works. But not necessary depending on the speed of the fastest drive, which may support both quite well.
The administrator has disabled public write access.

Spread media across hard drives 22 Dec 2017 09:24 #92661

  • Andreas Kiel
  • Andreas Kiel's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 990
  • Thank you received: 167
  • Karma: 32
While reading that another thread came into my mind:
www.fcp.co/forum/4-final-cut-pro-x-fcpx/...-avoid-missing-media

So I created a little "Integrity Checker"
A free beta (as Christmas gift) can be downloaded here:
www.spherico.com/filmtools/X-Files/X-Asset/XAU.dmg

There is no manual or help published yet but usage is simple.
Drag a fcpxml to the apps icon or drag a library, event or project directly from the FCP browser to the app's main table.
All involved bundles and volumes will be listed, all assets (including status info) as well.
Now you can select a volume (probably the lib bundle's volume) and get a list of assets which are not on this volume. The list can be saved as tab-separated-text file.
Some additional features: show selected files in Finder or do a Spotlight search for missing files.
You also can save the current asset list and compare it against a remote user's list or a prior version.

Have fun
Andreas
Last Edit: 22 Dec 2017 09:29 by Andreas Kiel.
The administrator has disabled public write access.

Spread media across hard drives 22 Dec 2017 11:40 #92663

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1322
  • Thank you received: 275
  • Karma: 26
Andreas, thanks for this. Here are some examples to illustrate how useful this utility is:

Scenario 1: You have a portable drive containing a non-managed library and media set you want to send a remote collaborator. IOW the media is external, not in the library. You obviously don't want to ship the drive then find it didn't contain all the media, forcing you to ship another drive (because the assets are too large for file transfer).

While the FCPX library inspector will show you IF assets are on another volume, it does not show you which ones. There is no easy way to locate these. Andreas' utility does that quickly and easily. It ensures the drive you ship contains 100% of all media assets -- the first time.

True, FCPX can consolidate media from one non-managed location to another, then you're guaranteed that new media tree contains everything. That works for small cases. My current documentary library is 20 terabytes. I don't have the time or disk space to copy that to another array, simply because FCPX failed to provide any way to list what media files are on what volume. By contrast Andreas' utility works in real-world production situations, not just for "toy" libraries.

Scenario 2: At the start of post production, you and a remote collaborator both have hard drives with the same media assets. The remote editor needs to send you the library, event or project XML for color correction or audio. Unfortunately the remote editor has added more media assets but didn't put them all in a known location. They are fatigued, have been working 18 hr a day, and can't remember if every file was ingested from a designated media tree. Typically they send a project XML file, you load it, see red thumbnails and spend lots of time on the phone trying to isolate which files those are.

Andreas' utility allows the remote editor to simply send a library, event or project XML file which you then compare with your own version, and it flags any media file differences. IOW it is an FCPX library/event/project "diff" utility. It can compare a project to a library and find any media files referenced by the project that are not in the library and export a list of these.

Thus you can provide the remote editor a list of newly-added media files they need to send you -- all in a single step. No time spent on the phone tracking down missing files. You could also send your library XML to the remote editor and they could use the utility on their end to isolate these stray files before sending them.

I have previously spent precious hours In time-critical post production situations tracking down files. If a drive is shipped without containing all the media, this could easily cost 1-2 more days while the files are located and another drive is shipped.

If everyone exercised perfect procedures on media files, you wouldn't need this utility. But in the real world that does not always happen, so the utility helps in those situations. That is a lot of cases. It probably has additional uses I haven't even figured out yet. Thanks again to Andreas for taking time to create such a valuable tool.
The administrator has disabled public write access.

Spread media across hard drives 22 Dec 2017 14:27 #92667

  • Xavier Novembre
  • Xavier Novembre's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 42
  • Thank you received: 1
  • Karma: 1
Joema, did you give a look at Final Cut Library Manager ?
The administrator has disabled public write access.

Spread media across hard drives 22 Dec 2017 14:33 #92669

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1322
  • Thank you received: 275
  • Karma: 26
Yes I have it but was not aware it could "diff" all the media file references between libraries, events and projects.
The administrator has disabled public write access.

Spread media across hard drives 22 Dec 2017 14:45 #92670

  • Xavier Novembre
  • Xavier Novembre's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 42
  • Thank you received: 1
  • Karma: 1
joema wrote:
Yes I have it but was not aware it could "diff" all the media file references between libraries, events and projects.


that and much more ! they have a free version introducing the concept
The administrator has disabled public write access.

Spread media across hard drives 22 Dec 2017 15:00 #92671

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1322
  • Thank you received: 275
  • Karma: 26
I have a paid copy of Final Cut Library Manager but I don't see in the latest feature list the ability to (a) compare a local and remote library/event/project via XML and produce a "diff" report, nor (b) the ability to validate that each media reference in a library is present within a specified volume and reproduce a report of any which are not.
But thanks for the mention, I will examine the latest version more closely tomorrow.
Last Edit: 22 Dec 2017 15:02 by joema.
The administrator has disabled public write access.

Spread media across hard drives 22 Dec 2017 16:31 #92672

  • Xavier Novembre
  • Xavier Novembre's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 42
  • Thank you received: 1
  • Karma: 1
it does " b" showing you for each Library what is there and what is missing, which is the main point

. you can't copy or export that list : that's a big flaw.

"a" would be an other kind of app (but talk to the developers, they like to explore)
The administrator has disabled public write access.

Spread media across hard drives 22 Dec 2017 17:53 #92673

  • Andreas Kiel
  • Andreas Kiel's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 990
  • Thank you received: 167
  • Karma: 32
I see quite a big difference between X-Asset Utility and Final Cut Library Manager just from the approach.
X-Asset Utility only cares about assets. And only about assets mentioned in an XML (either from library,event or project), Final Cut Library Manager looks for libraries on the available volumes and analyse those based on these data.

While this would create somehow same kind of data (regarding assets) the data display and data handling could lead to different results.

Xavier Novembre wrote:
it does " b" showing you for each Library what is there and what is missing, which is the main point

It's not the "main point".
Joe's (b) cant't be handled with Library Manager. It's not about "online/offline" but "on my volume/not on my volume". "online/offline" in this case only is a subclass of latter criteria.
And this is different even if there would be no export of a list.
Regarding offline files X-Asset does a spotlight search for those as well.

Xavier Novembre wrote:
"a" would be an other kind of app (but talk to the developers, they like to explore)

Joe's (a) cant't be handled that easy with the approach using "real" libraries. Final Cut Library Manager is called Library Manager cause it handles Libraries.
You always have to have the local and the remote/other one. A project or event only won't work.

Anyway X-Asset is free for Christmas and use it if you want.
If it makes sense for you make a donation and look forward a version 1.1 ;-)

Andreas
Last Edit: 22 Dec 2017 18:28 by Andreas Kiel.
The administrator has disabled public write access.

Spread media across hard drives 19 Jan 2018 16:55 #93515

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2944
  • Thank you received: 391
  • Karma: 34
Final Cut Library Manager 3.5 has an option to export to CSV file, so you can track your media.
arcticwhiteness.com/blog/fclm350/
The administrator has disabled public write access.

Spread media across hard drives 19 Jan 2018 18:11 #93518

  • Andreas Kiel
  • Andreas Kiel's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 990
  • Thank you received: 167
  • Karma: 32
FCPX.guru wrote:
Final Cut Library Manager 3.5 has an option to export to CSV file, so you can track your media.
arcticwhiteness.com/blog/fclm350/
As I said above: XAU is not about just listing assets. It list assets by volumes to prepare a "move" for libraries/events/projects (or for verification)
Additionally it allows asset-comparison of 2 libraries/events/projects.

Final Cut Library Manager is a great app, it just doesn't cover these 2 points.
The administrator has disabled public write access.

Spread media across hard drives 19 Jan 2018 19:54 #93520

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2944
  • Thank you received: 391
  • Karma: 34
With the new CVS export, you can go into a spreadsheet or FileMaker Pro and do all sorts of things. I'd not write it off as not being able to help move stuff. I'd think it could do just that. I can compare lots of Libraries in a simple spreadsheet with a couple of CVS exports.
The administrator has disabled public write access.