The subgit-bitbucket plugin is mirroring each project happily.
When I tried to do this manually for subgit, when I tried to configure the second repository it told me subgit was already installed. I don’t have the instance any more to see the exact problem now.
Before I try to repeat the experiment I wanted to know if it is supposed to be possible, because I noticed that unlike some of the other subgit documentation, “import to gitlab” only seems to talk about one repository.
basically, both SVN Mirror and SubGit have the same functionality and work with the SVN the same way, so SubGit is definitely able to mirror those projects in case if SVN Mirror add-on has already set this way.
SubGit may report that it’s already installed if subgit configure is invoked against a repository that already contains SubGit metadata. So if, for example, you are trying to install SubGit in a Bitbucket repository already covered by SVN Mirror add-on, then such a message is expected. If, on the other hand, you are trying just to establish two different mirrors – one with SVN Mirror and another with SubGit – of a single SVN project or repository, then it’s perfectly possible, yet it’s not completely clear for me how had it happened a target SubGit repository already has SubGit metadata.
In fact, most of our documentation talks about one repository, not only “Import to GitLab”. The thing is that SubGit supports two different mirror modes, local and remote, the former of which is obsolete and not recommended for production use. The GitLab manual describes the remote mode where SubGit is installed into a single Git repository, mirrors a single SVN repository or project and the SVN repository is accessed remotely even it resides on the same computer – in contrast to SubGit’s local mode where SubGit is installed into the SVN repository, accesses the SVN repository and its database directly and mirrors all the projects in this repository at once and all the target Git repositories reside right inside the SVN repository. SVN Mirror, in its turn, is only able to run remote mirrors, just by its nature.
So if you are trying to mirror some SVN repositories/projects both with SVN Mirror add-on and with SubGit at the same time, then yes, it’s perfectly possible, even if SubGit’s mirror is planned to be of the local kind (yet I wouldn’t recommend using local mode). The subgit is already installed appears when the target repository (SVN or Git depending on the setup kind) already contains SubGit metadata; and the GitLab guide describes the remote setup which is essential for GitLab and Bitbucket and is the recommended mirror mode for all new mirrors.
Hope I managed to make it clearer a bit.
SubGit does not resume the synchronization automatically after the system reboot, indeed, it just has no means to add itself to the system autostart, so this should be done manually, by adding subgit install REPO or subgit fetch REPO in the system startup scripts for every mirrored repository, for example. Using those commands, actually, is the right way to start the daemon explicitly; but the daemon can also be started implicitly on a next push to a mirrored repository.
The shared daemon is definitely an appropriate way, too – in fact, it is the same SubGit daemon, but in a mode allowing to handle more than one repository. In a default SubGit setup (with no shared daemon) SubGit will start a separate daemon process for every mirrored repository; with the shared daemon it’s one process for several repositories. This approach has some benefits – it requires less resources for the daemon as it’s only one process – as well as some drawbacks – for example, if a single shared daemon handles too many repositories, then it may be required to extend the polling period. It’s up to you to decide which approach to use, they both are equal from the SubGit operations point of view only differing in the configuration; and I must note here that the shared daemon has no means to autostart either, so start the shared daemon after the system restart should be configuration manually, too.
It turns out the issue was that I’m using “sudo subgit register”, which results in git objects in the repository that are root.root, later on attempts to start subgit fail when they can’t create git objects because the hash.git/objects/nn/nn folder is not writable to them.
@ildar.khusainov I see the docs on the rest api, but could find nothing about actually setting up such a daemon.
It seems like you just need to give them all the same pidFile path (e.,g /var/run/subgit-daemon.pid) but it would be nice to be able to confirm this by reading the docs rather than having to ask :)
Well, I thought I had it working, but with 3.3.13 it only works with the last repository you install.
I set all 5 repositories to use /var/opt/gitlab/git-data/subgit/daemon/daemon.pid ( after creating the directory ). Then I ran subgit install on each of them to update their config, and then ran a subgit config on the last path I’d installed.
When I rebooted, I tried to subgit fetch --async $repos for all of them, and all but the last one failed.
So then I tried
for repo in $(get-repos); do
subgit install $repos
subgit fetch --async $repos
done
the flaw in this plan is … each install kills the already running shared daemon.
So I figure I’m missing something?
If this is unclear, I can try and set up a branch of the docker container I used for the svn+ssh issue?
It turns out the issue was that I’m using “sudo subgit register”, which results in git objects…
That is pretty strange since the register command is supposed to run on behalf of root since it needs to write to /etc but it’s not supposed to create any references in the repository, we’ll check this out.
I’m afraid setting the same pid for different repositories is not the correct way to set the shared daemon; in fact, the only needed option to configure it is the directory in the daemon “shared” section:
[daemon “shared”]
directory = PATH
Here is the procedure to configure several repositories with the same shared daemon:
This command will create repositories and SubGit configuration as the regular configure command, but will also add the shared daemon’s section in the config. It’s also possible to create configuration with the regular configure command, but add the daemon setting later:
In addition regarding this issue: I have tested versions 3.3.13 and 3.3.14 on Debian Linux and haven’t managed to reproduce the issue, SubGit commands work well even after the repository registration. The subgit register command being invoked on behalf of root (sudo/su) indeed creates a couple of file owned by root, but not in the Git’s objects directory, those are actually the registration process log (subgit-register--.zip) and subgit.key file in the REPO/subgit directory. In fact, this command does not perform any Git operations and thus does not affect Git objects at all, so I believe, those permissions on Git objects in the repository must have been set in some other way – by another process running on behalf of root, for example.
I see that it writes some config to the shared daemon directory, is there documentation on how I can convert my existing, running repos to shared daemon?
@ildar.khusainov per the above - I’ve modified the configs of the repositories to add the shared pidfile, which did not get seem to get created when I did this.
Once I managed to get it started, it sits spamming the log with complaints about the non-shared daemon.pid file not existing, and then deletes it just to make sure:
[2022-05-09 23:47:04.973][shared-daemon][30] About to release lock: Lock [file=/var/opt/gitlab/git-data/repositories/@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git/subgit/daemon.lock; memory=java.util.concurrent.locks.ReentrantLock@7a3bae57[Locked by thread Thread-11]; fs=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid]]
[2022-05-09 23:47:04.974][shared-daemon][30] Trying to release file lock on '/var/opt/gitlab/git-data/repositories/@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git/subgit/daemon.lock'.
[2022-05-09 23:47:04.974][shared-daemon][30] Released file lock on '/var/opt/gitlab/git-data/repositories/@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git/subgit/daemon.lock'
[2022-05-09 23:47:04.974][shared-daemon][30] Closed random access file '/var/opt/gitlab/git-data/repositories/@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git/subgit/daemon.lock'
[2022-05-09 23:47:04.974][shared-daemon][30] error while polling repository at '/var/opt/gitlab/git-data/repositories/@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git'
org.tmatesoft.translator.util.f: Failed to launch background translation process: timeout waiting for pid file '/var/opt/gitlab/git-data/repositories/@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git/subgit/daemon.pid'.
at org.tmatesoft.translator.util.f.b(SourceFile:27)
at org.tmatesoft.translator.process.r.a(SourceFile:65)
at org.tmatesoft.translator.daemon.j.b(SourceFile:513)
at org.tmatesoft.translator.daemon.j.a(SourceFile:379)
at org.tmatesoft.translator.daemon.j.a(SourceFile:230)
at org.tmatesoft.translator.daemon.j.a(SourceFile:193)
at org.tmatesoft.translator.l.k.a(SourceFile:92)
at org.tmatesoft.translator.l.k.a(SourceFile:53)
at org.tmatesoft.translator.l.k.call(SourceFile:17)
at org.tmatesoft.translator.daemon.P$1.call(SourceFile:124)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.tmatesoft.translator.daemon.P.run(SourceFile:77)
[2022-05-09 23:47:04.976][shared-daemon][30] scheduling task to run now
[2022-05-09 23:47:04.976][shared-daemon][30] tasks run complete
[2022-05-09 23:47:04.976][shared-daemon][30] about to be idle for 5481ms.
[2022-05-09 23:47:10.457][shared-daemon][30] about to run 1 tasks
[2022-05-09 23:47:10.458][shared-daemon][30] polling repository 2 of 5
[2022-05-09 23:47:10.458][shared-daemon][30] polling: /var/opt/gitlab/git-data/repositories/@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git
[2022-05-09 23:47:10.466][shared-daemon][30] repository at '/var/opt/gitlab/git-data/repositories/@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git' is idle, will poll
[2022-05-09 23:47:10.466][shared-daemon][30] Obtaining file lock on '/var/opt/gitlab/git-data/repositories/@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git/subgit/daemon.lock'.
[2022-05-09 23:47:10.466][shared-daemon][30] Obtained file lock on '/var/opt/gitlab/git-data/repositories/@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git/subgit/daemon.lock'
[2022-05-09 23:47:10.466][shared-daemon][30] Pid file does not exist: /var/opt/gitlab/git-data/repositories/@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git/subgit/daemon.pid
[2022-05-09 23:47:10.466][shared-daemon][30] Daemon info pid=-1; port=-1; address=; user=null from file '/var/opt/gitlab/git-data/repositories/@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git/subgit/daemon.pid'
[2022-05-09 23:47:10.466][shared-daemon][30] Pid file '/var/opt/gitlab/git-data/repositories/@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git/subgit/daemon.pid' deleted.
Configuring a running mirror to use shared daemon in fact as simple as configuring a new mirror with a shared daemon, the only needed thing is to add the daemon "shared" section and the shared daemon directory; the only difference is that the repository-specific daemon must be stopped prior to the reconfiguration, so the steps for a running mirror would be: