SVN files are not moved to GitLab

I want to move my SVN repository into GitLab through SubGit. I have followed the step of SubGit and got the following output:

Amits-MacBook-Pro:bin amitraj$ ./subgit install $GIT_REPO_PATH
SubGit version 3.3.10 (‘Bobique’) build #4368

Translating Subversion revisions to Git commits…

Subversion revisions translated: 563.
Total time: 215 seconds.

INSTALLATION SUCCESSFUL

You are using SubGit in evaluation mode.
Your evaluation period expires on August 4, 2020 (in 7 days).

Extend your trial or purchase a license key at https://subgit.com/pricing

But the files are not moved to GitLab. I also asked for subgit license file but I also have not received it yet. I filled the required form 28 July 2020.

Hello Amit,

could you please advise how did you find that the files were not moved to GitLab: namely, did you make this conclusion just by that you see in the GitLab UI or had you tried to clone the repository and files were not there? If the former case this is just GitLab UI issue, this condition can be cleared by the following command:

gitlab-rake cache:clear

it will clear the UI cache so the new files in the repository will appear in the UI.

If you have cloned repository and don’t see your files from SVN, then most probably your mapping configuration used for the import was not correct; this case will require a deeper investigation to resolve. For that, please collect and send us SubGit logs from the repository (they reside in /subgit/logs directory) – send us all the files that reside in that directory. Besides, please describe your SVN repository layout and what exactly from SVN you need to import to Git.

I tried to send you an email with the license key, but it looks like your mail server does not allows email from our domain. Please let me know another address I could sent the key to.

Thanks for your response. I am the same guy who has opened this ticket. I have sign-up with the same email-id but still did not get the email verification mail. So I have used a different id. You can send me the license file on ‘abhiamitraj@gmail.com’.

Please let me know where should I run the following command:

gitlab-rake cache:clear

Please find the following trace files related to it:
subgit-import-20200730-182456.zip (6.8 KB)

Besides, please describe your SVN repository layout and what exactly from SVN you need to import to Git.
There is one repository in which there are multiple folder and sub-folders are there. I need to move whole repository to GitLab.

Other trace files are as follows:
subgit-install-20200730-182022.zip (8.7 KB)
daemon.0.log (4.6 MB)

Other trace files are as follows:
subgit-configure-20200730-181431.zip (3.0 KB)
subgit-configure-20200730-181520.zip (2.3 KB)

The command
gitlab-rake cache:clear
should be run anywhere in the console. The only thing to take into account is that one has to run this command on behalf of ‘root’ system user.

If this doesn’t help, could you explain what you mean

the files are not moved to GitLab

i.e. what do you do to check whether the files are moved, what do you expect to see and what do you see instead?

This “gitlab-rake” command should fix GitLab UI if it’s only UI caches which is the issue.

Also I’ve looked into the logs attached, there’re exceptions:

Caused by: java.io.IOException: There was a problem while connecting to 192.168.7.46:22
at com.trilead.ssh2.Connection.connect(Connection.java:817)
at org.tmatesoft.svn.core.internal.io.svn.ssh.SshHost.openConnection(SshHost.java:225)
at org.tmatesoft.svn.core.internal.io.svn.ssh.SshHost.openSession(SshHost.java:153)
at org.tmatesoft.svn.core.internal.io.svn.ssh.SshSessionPool.openSession(SshSessionPool.java:85)
at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:122)

and

Caused by: java.io.IOException: Password authentication failed.
        at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:369)
        at com.trilead.ssh2.Connection.authenticateWithPassword(Connection.java:345)
        at org.tmatesoft.svn.core.internal.io.svn.ssh.SshHost.openConnection(SshHost.java:249)
        at org.tmatesoft.svn.core.internal.io.svn.ssh.SshHost.openSession(SshHost.java:153)
        at org.tmatesoft.svn.core.internal.io.svn.ssh.SshSessionPool.openSession(SshSessionPool.java:85)
        at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:122)

So there were issues with network connection to the SVN repository (“Host is down” error) and password authentication to SVN. Have you resolved those issues?

After looking at “subgit-install.0.log” you’ve attached I think I’ve found out why you see no files. This is because by default SubGit expects trunk/branches/tags structure for the URL you specify.

This is the best article to about your case
https://subgit.com/documentation/branches-mapping.html

I guess that chapter 5.1 Single Directory describes your situation. This corresponds to

subgit configure --svn-url svn+ssh://amitraj@192.168.7.46/home/svn/svn/language --layout directory path/to/repo.git

call. Unfortunately this option cannot be changed on the fly, so you have to do that from scratch (i.e. on a new empty repository without SubGit).

Thanks for your response. I am able to move my whole SVN repository into the GitLab. I am also able to register the provided license file. I just want to make sure that I have followed the correct steps. First I ran the following commands in the ‘subgit-3.3.10/bin’ directory:

./subgit configure --layout directory $SVN_PROJECT_URL $GIT_REPO_PATH
./subgit install $GIT_REPO_PATH

Then I ran the following command after putting the subgit license file in the ‘language.git/subgit/’ directory

sudo ./subgit register --key …/language.git/subgit/subgit.key …/language.git

Then I ran the following commands in the ‘language.git’ directory:

git remote add origin …/language.git
git push -u origin master

I have made some changes and commit in SVN repository. Then I ran the following command in the ‘subgit-3.3.10/bin’ directory:

./subgit import $GIT_REPO_PATH

After that I ran the following command in the ‘language.git’ directory:

git push -u origin master

Now I have 2 queries regarding this mirroring which are as follows:
I saw that when I updated the SVN repository, the GitLab repository is also updated after running two commands.
1. How can I do the same from the GitLab repository to the SVN repository means when I update the GitLab repository first and SubGit will update the SVN repository?
2. Can this mirroring be done automatically means without running any command (subgit import and git push)?

Thanks for your support in advance.

Everything you wrote until “subgit import” was correct. But you never need “subgit import” because “subgit import” is just for one time SVN->Git conversion, not for continuous synchronization.

Actually after “subgit install” you don’t need any command at all. Here’s why.

If you push to the Git repository, ‘pre-receive’ hook is automatically triggered and it sends to SubGit daemon a command to synchronize the change you’ve just pushed to SVN. No special command is required. If SubGit daemon is not running (e.g. if you restarted the server), the ‘pre-receive’ hook will start it and the daemon will stay in the background.

If you commit to SVN, SubGit doesn’t receives any notification automatically but SubGit daemon (if it’s running) checks for new changes in SVN once a minute (svn.fetchInterval option regulates this). So no special command is necessary again.

The only thing to do is to make sure SubGit daemon is running. You can always check it using ‘jps’ command which is provided together Java distribution.
$ jps | grep SubGitDaemon
12787 SubGitDaemon
This command (as well as SubGit daemon itself) should be run on behalf of ‘git’ system user (i.e. the same user as GitLab uses).

When you run ‘subgit install’, it starts the daemon automatically, so unless you restart the server. If you restart the server, it’s enough to either push to the Git repository to start SubGit daemon. If you have nothing to push, you can use this command

git push -q origin --delete refs/subgit/command/fetch

OR

git push -q $FULL_GIT_URL --delete refs/subgit/command/fetch

It’s a command to delete an artificial non-existing branch. This will trigger ‘pre-receive’ hook, which will start SubGit daemon. The daemon will interpret this as a command to perform SVN<->Git synchronization immediately.

Finally, you can use ‘subgit fetch’ command (on behalf of ‘git’ system user) for starting SubGit daemon after restarting the server. It also just starts SubGit daemon and performs SVN<->Git synchronization.

There’s a number of other ways to configure SubGit but the easiest thing to do is just to do nothing. If you push to the Git repository more or less regularly, you don’t have to worry about SubGit daemon.

So my answers to your questions are:
1. How can I do the same from the GitLab repository to the SVN repository means when I update the GitLab repository first and SubGit will update the SVN repository?
Just push to GitLab, this will trigger ‘pre-receive’ hook, which will tell the daemon to perform synchronization. Even if you change GitLab somehow without triggering the hook, wait a minute, and the synchronization will be performed automatically. If you don’t want to wait, push something to GitLab, e.g. a deletion of a non-existing branch.

2. Can this mirroring be done automatically means without running any command (subgit import and git push)?
Yes, this is already performed automatically in both directions, you never need to run “subgit import”.

I found an issue with the current architecture. While check in a code for one project, the developer has to check out the whole repository which contains all the projects, which is a dangerous situation.
Can I create one repository for one project after subgit translation where SVN architecture is as follows:
language/folder1/subfolder1/project1
language/folder1/subfolder1/project2
language/folder1/subfolder1/project3
language/folder1/subfolder2/sub-subfolder1/project1
language/folder1/subfolder2/sub-subfolder1/project2
language/folder1/subfolder2/sub-subfolder1/project3
language/folder1/subfolder2/sub-subfolder2/project1
language/folder1/subfolder2/sub-subfolder2/project2
language/folder1/subfolder2/sub-subfolder2/project3

Can you please let me know how can I implement this.

How I need to implement SubGit on SVN server so please tell me how can I define the SVN_PROJECT_URL parameter to implement SubGit

SVN_PROJECT_URL=svn+ssh://svn@192.168.7.46/home/svn/svn/language

Hello Amitraj,

Can I create one repository for one project after subgit translation where SVN architecture is as follows:

yes, it’s perfectly possible, you would just need to create separate mirrored repository for each project. For example, for the ‘project1’ project you would to set the URL as follows:

url = svn+ssh://svn@192.168.7.46/home/svn/svn/language/folder1/subfolder1/project1

If there are ‘trunk’, ‘branches’, and ‘tags’ directories inside the ‘project1’ directory, then the standard layout should be used; if the directory should be translated as a whole, then ‘directory’ layout should be chosen.
So for the standard layout use the following command to prepare the target SubGit repository:

subgit configure --layout std svn+ssh://svn@192.168.7.46/home/svn/svn/language/folder1/subfolder1/project1 $GIT_REPO_PATH

or for ‘directory’ layout:

subgit configure --layout directory svn+ssh://svn@192.168.7.46/home/svn/svn/language/folder1/subfolder1/project1 $GIT_REPO_PATH

The rest steps are the same, and the same procedure should be performed for all the projects.

How I need to implement SubGit on SVN server so please tell me how can I define the SVN_PROJECT_URL parameter to implement SubGit

Do I understand correctly you mean that you intend to install SubGit on SVN server? It is possible, but note that in this case the mirrored Git repositories will reside on the SVN server, too, as SubGit requires direct access to the target Git repositories directories. So in this case SubGit won’t be integrated with GitLab and the mirrored repositories won’t appear in GitLab.
As for URL – you can use file:// protocol to access the SVN repository in this case. So, URL for ‘project1’ may look like this:

 url = file///home/svn/svn/language/folder1/subfolder1/project1

How can I handle if there is space in the name of folder or subfolder? Auto syncing is not working at all, neither from SVN to GitLab nor GitLab to SVN. Please find the attached logs as well as the screenshot of the following command:

jps | grep SubGitDaemon

logs.zip (73.3 KB)

Hello Amitraj,

I didn’t find any errors neither in daemon log, not in install log, it looks like everything works well, the install log shows 574 revisions were fetched and the daemon log only shows periodic checks. So it’s not clear from the logs what the issue is.
As for the spaces in SVN folders names – this is completely supported and SubGit is able to handle such situations. Git does not support spaces in references names, so if SubGit detects spaces in SVN orders names, it replaces spaces with plus signs – that is, “SVN folder” becomes “SVN+folder” in Git.

I agree with you that I successfully migrated my one SVN project on GitLab repository. I made changes at SVN and committed it then I made changes on SVN and pushed it but in both the case I did not get the changes at either GitLab (SVN changes) or SVN (GitLab changes). I did not get the svn.fetchInterval parameter as well please guide me set this up so that auto-syncing will work properly.

When I ran the following command it gave me an error:

Amits-MacBook-Pro:bin amitraj$ ./subgit configure --layout directory svn+ssh://svn@192.168.7.46/home/svn/svn/language/Apple/JuBo Localhost /Users/amitraj/Documents/VBGitLab/JuBoLocalhost.git

error: expected arguments are: [–svn-url] SVN_PROJECT_URL [GIT_REPOS_PATH] or SVN_REPOS_PATH

error: Type ‘subgit help’ or ‘subgit help configure’ for more information.

But when I ran the following command it gave me the required output:

Amits-MacBook-Pro:bin amitraj$ ./subgit configure --layout directory svn+ssh://svn@192.168.7.46/home/svn/svn/language/Apple/JuBo\Localhost /Users/amitraj/Documents/VBGitLab/JuBoLocalhost.git
SubGit version 3.3.10 (‘Bobique’) build #4368

I agree with you that I successfully migrated my one SVN project on GitLab repository. I made changes at SVN and committed it then I made changes on SVN and pushed it but in both the case I did not get the changes at either GitLab (SVN changes) or SVN (GitLab changes). I did not get the svn.fetchInterval parameter as well please guide me set this up so that auto-syncing will work properly.

From this description it looks more like the changes were made in directories that are not included in the mapping configuration. As far as I found in the logs, the mirrored directory is PDDs:

svn+ssh://svn@192.168.7.46/home/svn/svn/language/PDDs

Could you advise please what are the changes you made in SVN? And same for GitLab, could you you advise what were the branch you made changes to in GitLab and what were the changes?

When I ran the following command it gave me an error:

That’s expected as you provided more arguments than SubGit expects:

./subgit configure --layout directory svn+ssh://svn@192.168.7.46/home/svn/svn/language/Apple/JuBo Localhost  Users/amitraj/Documents/VBGitLab/JuBoLocalhost.git

SubGit expects SVN_PROJECT_URL and [GIT_REPOS_PATH] while you provided three arguments: svn+ssh://svn@192.168.7.46/home/svn/svn/language/Apple/JuBo, Localhost, and Users/amitraj/Documents/VBGitLab/JuBoLocalhost.git. Embrace the URL with quotes:

./subgit configure --layout directory "svn+ssh://svn@192.168.7.46/home/svn/svn/language/Apple/JuBo Localhost"  Users/amitraj/Documents/VBGitLab/JuBoLocalhost.git

or replace the space with URL encoded space %20:

./subgit configure --layout directory svn+ssh://svn@192.168.7.46/home/svn/svn/language/Apple/JuBo%20Localhost Users/amitraj/Documents/VBGitLab/JuBoLocalhost.git

Either of these two approaches resolves the ‘subgit configure’ issue.

From this description it looks more like the changes were made in directories that are not included in the mapping configuration. As far as I found in the logs, the mirrored directory is PDDs:

Yes, your understanding is correct.

Could you advise please what are the changes you made in SVN? And same for GitLab, could you advise what were the branch you made changes to in GitLab and what were the changes?

Thanks for your response. I made a small change in a sheet whose name is ‘TGL PDDs.xlsx’ at GitLab and SVN simultaneously.

Is it possible to discuss this on a chat or something like that, so that we can solve this issue quickly?

Hello Amitraj,

I’m afraid we don’t have a support chat, we usually communicate here or over email.
What I meant asking about the changes you made was that which SVN directory were the changes made to and which Git branch were changes made to. ‘svn log’ and ‘git log’ output for the appropriate revision and Git commit (or the whole output for all commits) would be helpful. Please advise how did you do the changes in Git – did you make the changes in a working copy and push the changes to the GitLab repository or in some other way? Also, is the new Git commit being shown in GitLab UI?

Note, by the way, that if a change made to the same branch both in SVN and Git, then it will lead to a conflict. Yet this would be shown in logs (and also during the push to a mirrored repository) which is not true in this case, so it’s not the cause in our case.

Please find the whole SubGit folder from the below link:

https://www.sendthisfile.com/hOGQVARk5UOA2iZIPydUzbLj

I made a change in the working directory of SVN then commit through Cornerstone app. I made a change in the working directory of GitLab then commit through the following command:

git add . && git commit -m “GitLab to SVN” && git push -u origin master

Hello Amitraj,

could it be possible to provide us with SVN and Git logs from the affected repository?

I have created 7 repositories on GitLab and translated successfully but after that, I am not able to push the code on new repository. Please find the attached trace file.

logs.zip (396 Bytes)

I am getting the above error while pushing this project on GitLab. Please help me with this.