Can I exclude directorys after running the mirror?

Steve_Taylor · August 7, 2020, 2:14pm

I meant to exclude a certain directory before I took the mirror.

Am I able to retrospectively exclude it or do I need to delete everything and redo the mirror?

dmitry.pavlenko · August 7, 2020, 5:11pm

Yes, you can exclude files and directories using svn.excludePath and svn.excludeBranches option without retranslating everything from the beginning. To do that add

[svn]
  ...
  excludePath = *.iso
  excludePath = *.mkv
  ...

option under [svn] section. Then run

subgit install path/to/git/repository

to apply the change. Note that ‘excludeBranches’ excludes whole branches or tags from translation while ‘excludePath’ excludes some patterns (files and directories) inside every branch. It is not possible to exclude some certain file or directory for some certain branch.

Also note that there’s no way back. You can exclude some path, but you cannot “unexclude” them back later.

Also there’s svn.includePath option which excludes everything except specified patterns (the option can be used several times), it is similar to ‘svn.excludePath’. Again, you can only increase the set of excluded paths. Once you add ‘svn.includePath’ option and apply this change, you cannot add more ‘svn.includePath’ options later as this would decrease the set of excludedPath.

What about changing ‘svn.excludeBranches’ as well as ‘svn.trunk’, ‘svn.branches’, etc. : these rules can only be changed for branches that never existed in SVN yet. E.g. if you plan to add some branch to SVN you don’t want to see in Git, you add this rule to SubGit first, apply the change with ‘subgit install’, then add the branch to SVN. Otherwise you’ll need to use ‘–rebuild’ or ‘–rebuild-from-revision’.

Finally I should note that when using ‘svn.excludePath’ or ‘svn.includePath’, you don’t save network traffiic, only disk space. This is because of the way SVN works. The files are sent over network but are discarded locally, just before putting to Git. We have a hidden experimental option to optimize this behaviour. If this is your case, contact us again, I’ll tell you about it.

Steve_Taylor · August 17, 2020, 9:41am

Great, thanks.

Just so I can be absolutely sure being as there is no going back!

I have a directory which is:

Trunk/DLL

I want to exclude that and everything in it from every branch where it appears, so I need to use the command:

excludePath = /DLL

and then rerun the install command.

is that correct?

dmitry.pavlenko · August 17, 2020, 4:03pm

Hello Steve,
yes, that’s correct. I would recommend you to try that on a small test SVN repository first if you want to be sure.

Note that when you run install command, the ‘excludePath’ option will have an effect only on subsequent commits, not retrospectively. And you’re right that there’s no way back after that.

Also note that SVN is case sensitive, so I assume you have correct trunk/branches/tags/shelves options taking into account your case.

Steve_Taylor · August 17, 2020, 4:08pm

So, will the relevant directory simply vanish when we do a pull from the git repo or will it look as if it has been deleted from every branch?

My issue is that the directory is large and appears in almost every branch so I want to save space by not having it in the git repo (it is all generated anyway) it is still appears in previous tags then I am not saving any space when I do this.

dmitry.pavlenko · August 17, 2020, 4:35pm

It will not vanish. Exclude is not about vanishing, they are mostly about disentangling Git and SVN. I.e. after excluding from translation, pushes to Git changing that directory will not affect SVN while changes of this directory in SVN will not be translated to Git. Do the directory will stay after running ‘install’. Subsequent changes to it will not be translated.

Steve_Taylor · August 17, 2020, 4:37pm

I see.

Is there a way of having this directory not appear in the git repo at all? What if I redid the process from the beginning? Would the excluded folder still be present in the git repo?

dmitry.pavlenko · August 17, 2020, 4:41pm

Yes, if you do everything from scratch, it will not appear in Git because it will be excluded from translation from the start.

Also, if you have this directory added in the middle of a huge history, you can run

subgit install --rebuild-from-revision REV GIT_REPO

where REV is the revision that will NOT be re-translated (REV+1, REV+2, …, HEAD will be re-translated from scratch then). So if you try REV where /DLL never existed, it won’t appear in Git at all.

Steve_Taylor · August 17, 2020, 4:44pm

I think it has always been there. I am guessing I need to redo the mirror.

Do I need to delete the existing git repo first or can I make the changes to the config file and create the new mirror over it?

dmitry.pavlenko · August 17, 2020, 4:48pm

You can either run

subgit install --rebuild GIT_REPO

This command will re-trasnlate everything from scratch but it will automatically create a backup of your existing installation.

Or you can create a absolutely fresh new Git repository from scratch using ‘subgit configure’ + copying config and authors.txt and passwd files and then running ‘subgit install’.

Steve_Taylor · August 19, 2020, 7:41am

Hi,

I did the command:

subgit install --rebuild GIT_REPO

After changing the config file and everyone now gets this error when trying to push to master.

! [remote rejected] master -> master (unable to migrate objects to permanent storage)
error: failed to push some refs to ‘https://server.domain.com/root/therepo.git’

Is there anything I else I need to do after running rebuild?

It looks like new files in the objects/packs directory are being created as read only? Could that be it?

ildar.khusainov · August 20, 2020, 2:55am

Hi Steve,

the “install” command itself doesn’t require any additional actions, the repository should be ready for work after the command finishes its work.
From the error message “unable to migrate objects to permanent storage” it indeed looks like permissions issue. Could you advise please which protocol do developers use to commit to the mirrored repository, is that ssh? If so, have the repository permission been set groups-writable? If not, what is the user that runs SubGit, is that the same ‘git’ user that runs GitLab?

Steve_Taylor · August 20, 2020, 11:04am

Hi,

We use https to communicate with the git repo.

I think I did the rebuild command as root for subgit whereas the original mirror was as git. Could this cause the issue?

ildar.khusainov · August 20, 2020, 11:14am

Hi Steve,

yes, it’s definitely can be the reason. Moreover, if you ran the rebuild on behalf of ‘root’ then SubGit daemon probably us still running on behalf of root. So the solution is to stop the daemon in this repository:

subgit shutdown <REPO>

this command should be invoked by root.
Then, change the files owner to git in this repository:

chown -R git <REPO>

after that start the daemon on behalf of ‘git’ user:

subgit install <REPO>

Steve_Taylor · August 20, 2020, 1:39pm

Hi,

I have done the above. Now when I attempt git fetch I get the message:

$ git fetch
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500
fatal: the remote end hung up unexpectedly

Is there anything else I need to do?

ildar.khusainov · August 20, 2020, 2:01pm

Hi,

as far as I found, this error comes from Git client when it cannot deliver the data. It can be caused by large blobs, but I think in this case it may also be permissions issue. Could you please describe your setup in more detail: it is a GitLat repository on GitLab server and SubGit installed into this repository, right?

Steve_Taylor · August 20, 2020, 3:55pm

Hi,

I have done some more poking around and it looks like GitLab has filled the disk space on the server such that there is only 20k left.

I will sort this out and recheck the status of the git mirror.

Steve_Taylor · August 20, 2020, 4:21pm

Okay, now we have fixed the disk space explosion!

When I try to push to master I get this error:

To https://host.server.com/root/therepo.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to ‘https://server.domain.com/root/therepo.git’

ildar.khusainov · August 20, 2020, 4:24pm

Unfortunately, this message gives no clues on why the push has been declined. Is there other information in the message? Usually SubGit responds with the reason why the push has been declined. If there’s no information, please upload pre-receive hook and daemon logs.

Steve_Taylor · August 21, 2020, 10:20am

Okay, I have sorted the server side out.

I now get the message:

$ git push
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 363 bytes | 0 bytes/s, done.
Total 4 (delta 3), reused 0 (delta 0)
remote: Fetching revisions from SVN repository:
remote: up to date
remote: Sending commits to SVN repository:
remote: 3253f26 => r70067 trunk
remote: Sync completed successfully
remote: error executing git hookerror: hook declined to update refs/heads/master
To https://host.server.com/root/therepo.git
! [remote rejected] master -> master (hook declined)
error: failed to push some refs to ‘https://host.server.com/root/therepo.git’

Is this a standard thing to encounter? The commit still goes through and appears in svn, I just get this error message as well.