Hi,
I just configured subgit on a GitLab install for one repository which should mirror a remote subversion repository. The import from Subversion works great and the synchronisation from Subversion to GitLab works as well, but when trying to push to the cloned GitLab repository while the daemon is running I always get
Total 0 (delta 0), reused 0 (delta 0)
remote:
remote: SubGit ERROR REPORT (SubGit version 3.3.10 (‘Bobique’) build #4368):
remote:
remote: You’ve received this message because SubGit (http ://subgit.com/) is installed in your repository
remote: and an error that needs to be dealt with has occurred in SubGit translation engine.
remote:
remote: TEMPORARY ERROR:
remote: Failed to launch background translation process: timeout waiting for pid file ‘/var/opt/gitlab/git-data/repositories/@hashed/cc/3b/cc3b02a1d33c8770569ce864b253d728fed3b629d605bd911bd8d824e36302eb.git/./subgit/daemon.pid’.
remote:
remote: CURRENT STATE:
remote: Both Git and Subversion repository are open for pushes or commits.
remote: Your commit was not committed, but you may retry it.
remote:
remote: TO RECOVER:
remote: A) Address the problem if possible and then retry commit
remote: OR
remote: B) Run on the server
remote: $ subgit uninstall /var/opt/gitlab/git-data/repositories/@hashed/cc/3b/cc3b02a1d33c8770569ce864b253d728fed3b629d605bd911bd8d824e36302eb.git/.
remote:
remote: IMPORTANT: As soon as SubGit is uninstalled, both Git and Subversion repositories
remote: will become open, but no synchronization will take place.
remote:
remote: TO REPORT:
remote: Report an issue at http ://issues.tmatesoft.com/
remote: You may find logs on the server at ‘/var/opt/gitlab/git-data/repositories/@hashed/cc/3b/cc3b02a1d33c8770569ce864b253d728fed3b629d605bd911bd8d824e36302eb.git/./subgit/logs’
remote:
remote: THANK YOU!
To git.fh-aachen.de:carnot/carnot.git
! [remote rejected] testing -> testing (pre-receive hook declined)
error: failed to push some refs to ‘git@git.fh-aachen.de:carnot/carnot.git’
I can’t tell what’s causing this. After that error, when I check for the daemon process on the GitLab server, it’s gone (as well as the pidfile). What can I do to get the synchronisation working? Thanks!
Regards,
Daniel
Hi Daniel,
from the description it looks like you have some permissions issue so that daemon cannot access some file created during push. Most often this happens if, for example, the ‘subgit install’ command was invoked by one user (‘root’ most often) while the push start on behalf of another (‘git’, for example), so it would worth checking this out. The ‘subgit install’ command is supposed to be invoked on behalf of ‘git’ user that runs GitLab, especially to avoid permissions problems, so if you reveal that it was ran on behalf of ‘root’, then first stop the mirror:
subgit shutdown /var/opt/gitlab/git-data/repositories/@hashed/cc/3b/cc3b02a1d33c8770569ce864b253d728fed3b629d605bd911bd8d824e36302eb.git
change file owner in the repository:
chown -R git:git /var/opt/gitlab/git-data/repositories/@hashed/cc/3b/cc3b02a1d33c8770569ce864b253d728fed3b629d605bd911bd8d824e36302eb.git
and then start the daemon again, now on behalf of ‘git’ user:
sudo -u git subgit install /var/opt/gitlab/git-data/repositories/@hashed/cc/3b/cc3b02a1d33c8770569ce864b253d728fed3b629d605bd911bd8d824e36302eb.git
It also may happen that the daemon is slow to start and that causes the error (yet it does not seem to be very likely in this case); in this case, it may worth to increase daemon start timeout:
[daemon]
launchTimeout = 60
default value is 5, so it may worth to increase it to a minute or so.
If none of the above helps, please collect SubGit logs from the /var/opt/gitlab/git-data/repositories/@hashed/cc/3b/cc3b02a1d33c8770569ce864b253d728fed3b629d605bd911bd8d824e36302eb.git/./subgit/logs directory for analysis.
Hi Ildar,
the problem with the permissions is here, that GitLab runs in a docker container (with /var/opt/gitlab mounted to /data/gitlab/data), so the git user doesn’t exist on the host system. To get around this, I created the git user manually with the same uid from the docker container and ran subgit install as that user (of course after making sure the permissions are correct in the repo folder). I also tried with the launchTimeout setting as I found that in other issues as well, but it didn’t work.
I attached the log files:
daemon.0.log (21.7 KB) pre-receive-hook.0.log (93.3 KB)
I also made sure that subgit can find the repository at the path it expects it by symlinking /var/opt/gitlab/git-data on the host file system to /data/gitlab/data/git-data
Hi Daniel,
do I understand correctly you have GitLab running in the Docker container, while SubGit is running on your host machine outside of the container?
Yes, exactly that.
I’m afraid such a setup would not work, neither “GitLab in Docker/SubGit on host”, nor “GitLab in Docker/SubGit in another container”. SubGit is supposed to run on the same machine with GitLab, otherwise daemon runs on one machine (host in your case) while hooks start on another (in GitLab container), so hooks cannot communicate with the daemon process.
The best solution for GitLab and SubGit is to embed SubGit in the same container where GitLab runs. Another possible solution (well, a workaround, actually) is to remove SubGit hooks (rename them, for example). In a normal setup, SubGit uses two parts to run the mirror – SubGit daemon and hooks. Daemon runs periodic checks and translates changes from SVN when new revisions appear there. Hooks are being invoked when a push comes to a mirrored repository; hooks first check the SVN repository for changes (and downloads changes to Git if needed) and then send the newly pushed Git commits to SVN. So, the idea is to remove the hooks, so they are not being triggered when a push comes and thus push will succeed. However, it won’t be translated to SVN immediately it will only be sent later, during the periodic daemon check. This may lead to conflicts when, for example, a new revision is created in SVN at the same time push came in Git. The probability of such conflict may be decreased by decreasing svn.fetchInterval – it is the time between daemon checks, it’s 60 seconds by default – but such a probability will exist anyway.
Am I correct that the subgit daemon listens on a tcp port similar to 127.0.0.1:43241? Would it be possible to add a feature to subgit to be able to let it listen on a different interface so that it could be reached from inside the GitLab container? And while I am at it, would it be possible to add a foreground option to the daemon process so that it can be started in a container as well?
and to reply directly to your last question, what exactly do I have to remove to disable the hooks, just delete the custom_hooks folder?
Hi Daniel,
the daemon listens that port, but setting it to listen on an externally available network interface would not be a solution, the daemon and hooks processes should be on the same machine to work correctly anyway.
To disable the hooks it would be enough to rename the hooks files (or just remove them if you will).
All right, for now I’ll disable the hooks then. Please think of a solution in a way that subgit can run in its own docker container next to gitlab (in its own docker container). Will be especially interesting with GitLab running in Kubernetes, at least there subgit will need to be running inside a docker container. And running GitLab in Kubernetes is also the way gitlab.com is hosted by GitLab itself, so you should think of supporting that.