LFS objects

SubGit
#1

When setting up a one way mirror from SubGit to SVN how does SubGit deal with LFS objects?

#2

Hello,

SubGit does not support LFS and does not recognize the LFS pointer files in the repository treating them as regular files.

#3

Hi,

Just to confirm, we are setting up a “GitHub to SVN mirror” following your documentation. Are you saying that Git LFS files will be written as-is to SVN? Note that we have the LFS source setup and the LFS files are actually downloaded.

Thanks

Ali

#4

Hi Ali,

I meant that SubGit does not support using LFS right in the SubGit repository, it’s not able to work with the LFS storage. If you mean that you have LFS enabled in GitHub repository and the intend is to import changes from GitHub to SVN as per chapter 1.5 of the guide, then it may work, indeed, yet the changes should be pushed without use of LFS from the working copy to the SubGit repository.

#5

We are doing a one way sync from GitHub to SVN. My understanding is that the synchronization script will first do a git fetch and that should automatically take care of resolving LFS files. Will subgit fetch work in this case?

I have requested a Real-Time support session to make sure we do this properly.

#6

Hi Ali,

do you mean the script and the fetch configured right in the SubGit repository as it’s described in the chapter 2.4 of our GitHub guide? If so, then I’m afraid that no, this approach will not be working with LFS correctly.

#7

Correct. I think I will need to prevent using LFS for new files.

I have not heard from you regarding the support session.

#8

Hi Ali,

I am sorry for delaying the response to your request for the support session. As a matter of fact, we’re still unsure if we could be of any help.

You’re correct that you should stop using LFS for new commits. We recommend using Partal Clones instead. That prevents clogging the repository with large files.

However, after you stop using LFS, SubGit will only be able to sync your new commits to SVN correctly (with files). All older commits will still have LFS pointer files transferred to SVN instead of the actual files. We don’t have any means to convert that.

If this scenario works for you, we can schedule an online support session next week.

#9

Hi Dmitry,

Let me give a bit of background. We currently have a monolithic SVN repository for multiple projects that has 3rd party dependency dlls checked in source control. Ideally we would move those dlls over to an artifact repository and use nuget to fetch dependencies on the fly. So our repository will be free of any large files and there’s no need to use Git LFS or partial clone.

This is a change I’m planning to make after moving our development to Git. For now I’m doing a straight up conversion from SVN to Git and would like to keep a one way mirror from Git to SVN as a backup and that’s where Subgit comes in.

For all the large files in our current SVN history I will convert them to LFS as a one time conversion. Once we transfer to Git we will NOT allow any new LFS files or large files to be checked in Git. Existing LFS files can only be deleted and not modified.

Our conversion process will be as follows:

  1. Use Subgit to do one time conversion from SVN to Git
  2. Use Git LFS to migrate large files to LFS
  3. Start mirroring new commits from Git to SVN. No LFS in new commits
    Is it possible for subgit to handle this workflow? That is, after step 2 I would like Subgit to think that Git and SVN repos are in sync and start mirroring new commits only. It should not have to deal with LFS.

Please let me know if you think you’ll be able to deliver this.

Thanks
Ali

This e-mail, including any attachments, may contain information that is confidential and legally privileged. Any distribution, copying, forwarding, dissemination or reliance upon any of the information in this e-mail, or its attachments, by anyone other than an intended recipient is strictly prohibited. If you believe you received this email in error, please notify the sender immediately by reply email and permanently delete this message, including any attachments.

#10

Hi Ali,

Thank you for providing more details on your current setup and your goals on SVN to Git migration.

When you move large files from a Git repository to LFS throughout the repository history, you rewrite the hashes of old commits with new ones because the content of the new commits is different (large files are replaced with LFS pointer files). That breaks the SVN revision <=> Git commit mappings created by SubGit at step 1. As result SubGit won’t be able to start the mirror at step 3.

I think there are the following options available for you:

  1. Migrate your SVN project to a Git repository with SubGit mirror leaving all the large files as is in a Git repository. Unfortunately, the resulting Git repository is going to be heavy-loaded with all the large files now stored within your SVN repository. That means a full clone of that Git repository is going to take a lot of time and disk space. You can workaround this problem by using partial clones on the Git users’ side.

  2. Migrate your SVN project to a Git repository with SubGit mirror excluding all the large files from the sync. You can use excludePath option for that. The resulting Git repository is going to be light-weight, but the DLLs are going to be missing in that Git repository. I guess you won’t be able to build the project from the older Git commits, but you can use your SVN repository as an archive for these purposes.

  3. This is a weird option: you can convert your SVN project to a Git repository, run a script that moves all the large files into LFS and then convert resulting Git repository to a new SVN repository. This new SVN repository is going to differ from the original SVN repository due to LFS pointers being put instead of the original DLLs. At least this new SVN repository will be able to fetch changes from the Git repository via SubGit mirror.

Unfortunately, these are the limitations of SubGit with regards to LFS. To be honest, I think the second option looks a bit better than the others, but you definitely understand your setup and your migration goals better than us, so at the end of the day the choice is yours.

Hope this information is helpful. Feel free to ask any additional questions or share other ideas on your SVN to Git migration, we’d be glad to help you.

#11

Thanks for your suggestion, one more question with solution 2:
excludePath =
what if I have more than one file or file type to exclude?
such as .bat .dll .exe
or only want to exclude the only specified name of a.bat & a.exe, but let other b.bat… remain.

#12

Hello,

the svn.excludePath and svn.includePath directives support wildcards and recursive/non-recursive patterns that allows flexibly configure the exclusions. For example, all the files of a particular type can be excluded as follows:

[svn]
    …
    excludePath = *.bat
    excludePath = *.dll
    excludePath = *.exe

The files can also be referred by names, also with wildcards supported:

[svn]
    …
    excludePath = a.bat
    excludePath = a.exe
    excludePath = a*.dll

It is also possible to exclude files in a particular directory and keep files of the same type in another directory and many more combinations. Here is our guide for the paths options:

TMate SubGit: Branches and tags mapping

It describes all the exclusion rules, hope it will be useful.

1 Like
#13

Got it, great thanks.