Welcome, Guest
Username: Password: Remember me

TOPIC: Importing from SD card

Importing from SD card 30 Jan 2018 14:44 #93783

  • Dinky
  • Dinky's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 52
  • Karma: 0
Hi ,

Recently I have started to get some problems when importing video material from SD card. In my workflow I used to copy the MTS files off the SD card onto a hard disk folder structure so to keep all my source files. This has always worked fine, but recently there have been situations where a clip from the SD card was broken into two MTS files. This in itself would not be a problem, because I should be able to concatenate the two when editing in FCPX, except the audio in the last second or so of the first MTS is not there. In case of music recording this is a pain in the neck.
Funny enough when I import the clip straight from the SD card into FCPX, it is and remains a single clip. This, in my view, rules out the possibility of a problem in the camera. Also when I drag & drop the AVCHD file into e.g. VLC, I see the two MTS files appear. This specifically seems to be the case in longer clips (> 10 minutes)
Importing directly from SD card is not an option for me, because I use the concept of leaving the source files in place and only use aliases to them in the FCPX library. Importing from SD does not support this and it will always copy the files to Optimised Media. Also I noticed Image Capture does not recognise SD cards anymore, so I can't use that for import...
Is there a way to copy the files from the SD card without these problems ?
The administrator has disabled public write access.

Importing from SD card 30 Jan 2018 15:55 #93787

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2829
  • Thank you received: 372
  • Karma: 34
"and it will always copy the files to Optimised Media"

No, not always, only if you specify for your media to be Optimized.
The administrator has disabled public write access.

Importing from SD card 30 Jan 2018 18:03 #93788

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1268
  • Thank you received: 263
  • Karma: 22
Dinky wrote:
...copy the MTS files off the SD card onto a hard disk folder structure so to keep all my source files. This has always worked fine, but recently there have been situations where a clip from the SD card was broken into two MTS files. This in itself would not be a problem, because I should be able to concatenate the two when editing in FCPX, except the audio in the last second or so of the first MTS is not there. In case of music recording this is a pain in the neck...Importing from SD does not support this and it will always copy the files to Optimised Media. Also I noticed Image Capture does not recognise SD cards anymore, so I can't use that for import...

This is a clip-spanning issue which can happen if the files from tree-oriented media are copied "bare" away from the folder tree and imported. Metadata required for clip spanning is in the folder tree. Whether you have problems can be unpredictable and vary based on camera firmware, NLE version, etc.

However clip spanning is not the only concern. For AVCHD you should never, ever copy the MTS files out of the bundle and import them "in place". There is a significant I/O issue in FCPX whereby if AVCHD .MTS files are imported "in place" it will degrade library performance significantly -- even if only a small % of the files are that type.

The solution is either import from the AVCHD bundle which re-wraps them on import, or externally re-wrap them with EditReady then import using "leave files in place": www.divergentmedia.com/editready

That will avoid the I/O problems, make the import faster, and clip spanning will work using only the re-wrapped files since EditReady embeds the required metadata.
The administrator has disabled public write access.

Importing from SD card 30 Jan 2018 18:22 #93789

  • Axel
  • Axel's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 489
  • Thank you received: 61
  • Karma: 10
Dinky wrote:
--- but recently there have been situations where a clip from the SD card was broken into two MTS files. (...) This specifically seems to be the case in longer clips (> 10 minutes)


'Spanned clips'. The most notable tip of the iceberg of all real and potential problems you're inviting with your method of copying naked ('orphanized') AVCHD clips to your drive. The command to stitch the two (or more) files seamlessly lives in another subfolder - one you deemed useless and didn't copy.

The MTS files are not the complete clip. Not complete = not native.

FCP is notoriously much slower with these castrated files.

Dinky wrote:
Importing directly from SD card is not an option for me, because I use the concept of leaving the source files in place and only use aliases to them in the FCPX library.


Okay. But why?
You may have a valid reason. I'm just curious what your's is.


Dinky wrote:
Importing from SD does not support this and it will always copy the files to Optimised Media.


Like FCP.guru said, that's not the case. You must have looked into the library bundle and detected the ".mov". The imported clips are wrapped in a Quicktime-container. This container contains everything that on the original card was spread over the other folders. And the video and audio as well.



Dinky wrote:
Is there a way to copy the files from the SD card without these problems ?



Editready or Kyno. Kyno (1 month free trial) i.e. lets you preview your clips directly from card, subclip them (if needed), rename them (if needed), tag them (if needed), rewrap and/or transcode them (to ProRes) and copy those to a destination of your choice.

The resulting files are all self-contained. You can import them to FCP and leave files in place. It does the same FCP does, but externally.
The administrator has disabled public write access.

Importing from SD card 31 Jan 2018 09:42 #93798

  • Dinky
  • Dinky's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 52
  • Karma: 0
Thanks guys. I was not aware of the potential problems are of removing the metadata files. The idea behind this was to keep my video files in the best possible quality. I knew editing MTS files required a transition to another format but things like clip spanning are new to me.
I could now use EditReady, as suggested, or use FCPX to do the job. In FCPX I noticed that importing AVCHD will always copy the clips to Optimised Media. The option to leave the files in place is greyed out. So what should I do if I want to keep all my video files in the best possible format for future use ? Would it be sufficient to keep the AVCHD's ? I don't want to keep all the optimised media for ever taking up a lot of space. However if I delete the optimised media after I'm done with a project and later want to change something after all, all clips in the project will likely be shown as "missing" and I would need to re-import and relocate these clips. So I'm looking for the best workflow and way to keep my video files.
Any advice would be appreciated
Last Edit: 31 Jan 2018 10:29 by Dinky.
The administrator has disabled public write access.

Importing from SD card 31 Jan 2018 11:05 #93800

  • Dinky
  • Dinky's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 52
  • Karma: 0
Additional info
I purchased EditReady to see if that would help me and although it creates editable files from MTS files, the resulting files are huge. One of the reasons for keeping the MTS files was to preserve storage and at the same time keep the highest possible quality. Recreating optimised media in case I wanted to reuse the files I took for granted and these could be deleted if necessary
I am certainly not going to keep the enormous files created by EditReady. E.g three MTS files, forming a single clip, take up 5GB and result in a 45GB MOV file
So I guess keeping the AVCHD for possible future use and (re)creating optimised media whenever I need to edit the video looks like the only viable option.
The administrator has disabled public write access.

Importing from SD card 31 Jan 2018 11:13 #93801

  • Axel
  • Axel's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 489
  • Thank you received: 61
  • Karma: 10
Dinky wrote:
In FCPX I noticed that importing AVCHD will always copy the clips to Optimised Media.


Check this FCP preferences pane:

preferences.jpg


Dinky wrote:
The option to leave the files in place is greyed out.


Meant as a fail-safe behavior. With FCP, you can plug in the card, hit import and immediately start editing. The copying of the card's content is done in the background. Some complain that FCP should be able to detect if the source is the physical card or already a copy on the drive. Arguable, but that's just the way it is.


Dinky wrote:
So what should I do if I want to keep all my video files in the best possible format for future use ?



I'd say this depends on your shooting ratio. If you have recorded 64 GB AVCHD, but you find later on that you will use, say, only 5 GB of that, and if that's typical for you, then you might consider
a) Editready and the like to extract the "favorites" from the card beforehand.
b) Import selected with FCP X, but to a folder outside the library:

storagelocation.jpg



Dinky wrote:
Would it be sufficient to keep the AVCHD's ?



Well, the "MTS" are NOT the native AVCHDs!
AVCHD is defined as a *bundle*, in which additional information is stored (see spanned clips).

If 30 % of all clips of a 64GB SD card are useable, you can forget all this nonsense. You should backup the whole card to a USB3 stick (coming to think of it, USB2 would suffice too) using the FCP X camera archive (import window down left). And afterwards import selections from there.

I think you have to get one obstacle out of the way: shortage of storage space. It has become so ridiculously cheap! Especially for AVCHD. You can archive 3 hours of original footage for the price of a Starbucks coffee with a muffin. Crazy if you fume about wasted storage. My 2 cents.


Dinky wrote:
I don't want to keep all the optimised media for ever taking up a lot of space. However if I delete the optimised media after I'm done with a project and later want to change something after all, all clips in the project will likely be shown as "missing".



This must be a misconception on your side. FCP creates Optimized Media only additionally to Original Media! On the "lot of space" - remark: you are storing the media externally, not on your precious system drive, aren't you?
Last Edit: 31 Jan 2018 11:16 by Axel.
The administrator has disabled public write access.

Importing from SD card 31 Jan 2018 11:59 #93802

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1268
  • Thank you received: 263
  • Karma: 22
Dinky wrote:
... In FCPX I noticed that importing AVCHD will always copy the clips to Optimised Media. The option to leave the files in place is greyed out. So what should I do if I want to keep all my video files in the best possible format for future use...I purchased EditReady to see if that would help me and although it creates editable files from MTS files, the resulting files are huge.

As Axel said, importing from AVCHD will only created optimized media if that FCPX preference is set. You don't need that. Optimized media does not help video quality, only performance. AVCHD by definition is no higher than 1080 because the spec doesn't support that. Almost any Mac can edit 1080 content without using either optimized or proxies.

You also normally don't need background rendering, so you can go ahead and turn that off as well. Leaving it off will save space from new render files being constantly generated.

In EditReady you need to select the preset "Rewrap". That maintains the same file size and conversion is vastly faster because it does not transcode. One complete, *those* files can be imported in place.

For AVCHD I only keep the re-wrapped versions that EditReady makes. That way they will immediately sync up with your library if I later need to edit it.

How much to keep for archival purposes and in what format is a personal choice. I keep all the original camera files my documentary team shoots, with the exception of AVCHD where I keep the re-wrapped files. We haven't shot any AVCHD in four years, so it's really a non-issue for us but that's what I do.

In addition I create a separate managed library which only contains the project (aka timeline) and all referenced media. To do this create a new library and within FCPX drag/drop the project to that and answer "yes" to copying the media. Then you consolidate that library in the Inspector. This way I absolutely know the media for the timeline will be available and there will never be any relink problems. It is partially redundant to keeping the original files, but as Axel said, storage is cheap and these files aren't that big (relative to 4k or ProRes).
Last Edit: 31 Jan 2018 12:01 by joema.
The administrator has disabled public write access.

Importing from SD card 31 Jan 2018 15:20 #93804

  • Dinky
  • Dinky's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 52
  • Karma: 0
When importing an AVCHD, be it from SD card or after copying the AVCHD folder to hard disk, the option to leave files in place is greyed out and FCPX will always copy the files to optimised media.
The administrator has disabled public write access.

Importing from SD card 31 Jan 2018 15:27 #93805

  • Dinky
  • Dinky's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 52
  • Karma: 0
joema wrote:
Dinky wrote:
...
In EditReady you need to select the preset "Rewrap". That maintains the same file size and conversion is vastly faster because it does not transcode. One complete, *those* files can be imported in place.

For AVCHD I only keep the re-wrapped versions that EditReady makes. That way they will immediately sync up with your library if I later need to edit it. .
Thank you, this is what I was looking for
The administrator has disabled public write access.

Importing from SD card 31 Jan 2018 19:13 #93812

  • VidGreg
  • VidGreg's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 931
  • Thank you received: 185
  • Karma: 28
Hi Dinky
There is some really good info in this thread especially about AVCHD files.

Axel already answered your ? about why the "Leave Files In Place" is greyed out when on a camera card and the same goes for exact copies of the card like in "Camera Card Archives" or Finder copies…
"Meant as a fail-safe behavior. With FCP, you can plug in the card, hit import and immediately start editing. The copying of the card's content is done in the background. Some complain that FCP should be able to detect if the source is the physical card or already a copy on the drive. Arguable, but that's just the way it is."

Okay this is for all the folks that reuse their camera sd cards (all of Us!) so that we won't work off the card editing then erase card thinking it is already imported because we can see and edit in FCPX. Do make sure that background tasks are finished importing the files before quitting FCPX. FCPX will pause background tasks if you are concurrently editing so make sure it is finished importing. FCPX will only resume background tasks after 5 seconds of inactivity in FCPX, so imports can take a long time.

Like others, I keep multiple copies of original footage, first using a "Camera Archive" to an external, then import to project Library on a different external drive, then everything gets copied to another back-up drive using Carbon Copy Cloner. Even have a copy or 2 on TimeMachine. NOT Paranoid AM I???? :dry:

I guess in addition to all the other great advice, you might check out" Worx4 X" app that allows you to copy your project media files that are trimmed to just your project timeline along with some additional "Tops and Tails" of clips and creates XML files. Don't own it, but will. In App Store. Or do as Joema says about duplicating timeline to new project.

Hope this Helps, Greg
The administrator has disabled public write access.

Importing from SD card 31 Jan 2018 19:54 #93813

  • Dinky
  • Dinky's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 52
  • Karma: 0
Thank you all for your invaluable contributions. I am very sorry I was not aware of all this years earlier, but there you go. My new way of working started today :)

And, VidGreg, you are not paranoid at all. Your videos are to valuable to accidentally lose. Next to Carbon Copy Cloner I use an online backup service as well, just in case....
Last Edit: 31 Jan 2018 19:56 by Dinky.
The administrator has disabled public write access.