We recently upgraded to subgit 3.3.14 and are observing 100% CPU usage after a few hours of running the background job. We’re syncing 3 repositories with Github.
Platform is Ubuntu 20.04 and java -version reports the following.
openjdk version “11.0.15” 2022-04-19 LTS
OpenJDK Runtime Environment Corretto-11.0.15.9.1 (build 11.0.15+9-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.15.9.1 (build 11.0.15+9-LTS, mixed mode)
Let me know if you any more information.
Hello Ravi,
could you please advise how have you upgraded SubGit, have you run subgit install after SubGit’s binaries were upgraded in the system? What exactly do you mean saying “the background job” – is that about SubGit daemon or do you mean an initial import of a new mirror? What exact SubGit daemon generates that high load? I mean, if you have three mirrored repositories without shared daemon, then there should be three SubGit daemon processes in the system, so the question is if that is one of them that generates that hight load or they all show the same behaviour? Could you please collect all SubGit logs from a repository which daemon eats too much CPU (or from all three if all daemons behave the same way)?
Hi Ildar,
Yes, I ran subgit install.
By background job I mean the SubGit daemon. All 3 daemons generate high loads. Is there a way to share the logs securely as they seem to have all the settings logged in there?
Hi Ravi,
we usually ask to share logs right here attaching them to a topic.
If sharing here is not acceptable, the logs can be shared through a cloud service providing us with the link to download.
I can upload those to a cloud service but can you give me an email address I can send the link to?
Any update on this?
Hi Ravi,
we haven’t received any logs over email yet, have you sent them already?
Yes, same day.
Turned out it landed in spam as we didn’t notice it, sorry about that.
I have just requested the access to the files, could you please confirm?
Done. Please check.
Hello Ravi,
thanks, I managed to download the logs.
It looks, however, that you only included daemon logs, but we would also need both hooks and install logs, too, could it be possible to share those logs as well?
As for the issue – it looks like in all three repositories SubGit is unable to access the SVN server, the connection attempts fails with the following error:
svn: E210004: Handshake failed, data stream ended unexpectedly
org.tmatesoft.translator.util.f: svn: E210004: Handshake failed, data stream ended unexpectedly
It looks that SubGit is just unable to establish the ssh connection for some reason, could you please check what is in ssh logs on the SVN side? Also, could you please double-check if subgit install was performed for these particular repositories after SubGit binaries upgrade?
I’ve uploaded the other logs as well. Yes, I did run subgit install after the upgrade. It’s weird that SubGit is unable to connect to the SVN server as it’s the same machine. What I’ll do is stop the daemon, remove the logs and restart the daemon so we don’t have any stale messages.
In the subgit/logs directories I have the following files. Is that correct? I haven’t copied these here. I suppose this was done by Subgit install?
subgit-configure-20220114-070850.zip
subgit-install-20220527-072408.zip
subgit-register-20220114-071704.zip
subgit-install-20220114-071356.zip
subgit-install-20220607-025238.zip
Sorry, Ravi, but I didn’t get, what exactly was done by SubGit install?
As for the logs, it would be great to have at least latest install log from every repository.
Sorry, what I meant is I found these zip files inside the logs folder. Not sure if they are meant to be here. So I thought may be Subgit places them here for some reason.
I will capture latest logs from all directories.
Hello Ravi,
ok, thanks, it would be very helpful to have those logs.
Meanwhile, could you please try to edit SubGit start-up script and replace the following snippet:
-Dsvnkit.ssh.client=apache
with this one:
-Dsvnkit.ssh.client=trilead
Save the change and run subgit install against the mirrored repositories once again.
Installation fails…
SubGit version 3.3.14 (‘Bobique’) build #4433
About to shut down background translation process.
Background translation process is not running.
Jun 10, 2022 10:06:49 AM org.tmatesoft.svn.core.internal.io.svn.ssh.SessionPoolFactory create
WARNING: Using the obsolete trilead ssh client implementation, consider switching to apache by using -Dsvnkit.ssh.client=apache
INSTALLATION FAILED
error: svn: E210002: There was a problem while connecting to xxx :22
error: There was a problem while connecting to xxx :22
error: Key exchange was not finished, connection is closed.
error: Cannot negotiate, proposals do not match.
error: Unexpected error has occurred; please report along with the logs (’/home/git/subgit-install-20220610-100649.zip’)
thanks for checking and sharing.
It looks that the issue is on the SVN side then: SubGit 3.3.14 uses another SSH implementation by default, so I though it might have been restored with the old library that was in use in previous versions of SubGit. Apparently, it didn’t work, the issue is present with either of ssh implementation, so it would worth checking the SVN server for the issues.
Any idea what to check?
We’re using svn version 1.13.0 (r1867053)
compiled May 12 2022, 20:47:08 on aarch64-unknown-linux-gnu
Old Subgit worked fine.