Hi Anton,
I’m afraid we don’t have a resource where it could be possible to upload logs privately. Could it be possible to share them using some cloud service or sending them over email if they are not too big?
Hi Ildar.
Yes, email would be fine.
I got an automatic answer from your mail server that this address does not exist.
Yes, I recognise the mistake. I’ve sent our logs via email.
Hello Anton,
thank you for the files, we have received them.
Judging from the logs, the install process is being interrupted by two main reasons: a file permissions error like java.nio.file.AccessDeniedException: /mnt/gerrit/review_site/git/manage.git/refs/subgit/unsynced/heads/master/56.lock and a communication problem com.syntevo.svngitkit.core.b.i: svn: E210004: Handshake failed, data stream ended unexpectedly. The first one is most probably caused by wrong files ownerships, as we discussed above, so it would worth changing the files ownerships and permissions in the repository.
As for the second issue – it looks to be coming from the SVN server side, such an error usually appears when the server closes/breaks the connection unexpectedly. However, it’s not clear from the logs why exactly the SVN server did that, so it would probably worth checking logs on the SVN side to find out the cause.
As for the daemon issue – it looks to be caused by the OOME errors. The daemon does not look to hang, it fails with out of memory error and restarts the process over and over again, but every attempt fails with the OOME. It looks that heap size has already been increased and almost all relevant SubGit configuration options have already been set, but one that allows SubGit to use Git binary to stream data that can significantly decrease memory and disk space consumption. Here is how it can be set:
[core]
...
gitPath = /path/to/git/binary
As usual, subgit install must be invoked to apply the changes. I suggest applying this change and check out if it eliminates the OOMEs/GC overhead limit errors during the installation and daemon operations.
Hi Ildar.
Thank you for your answer.
I’ve added the option that you suggested in the configuration file. Will see if it helps.
However, I don’t understand one moment. Those failures that you found appear only during the installation stage after each new SVN revision. It works perfectly as soon as the Subgit finishes the installation and demonizes himself.
Hi Anton,
the errors that I found in the install logs didn’t happen during all the install attempts, some of them eventually finished with success starting the daemon; but the fact the errors arose indicates that some problems are present, at least sporadically, and they affect the install process.
As for the daemon – it does not seem to be working perfectly, the log shows it fails with OOME over and over again trying to get SVN log.
Hi Ildar.
We increased the memory limit to 32g, and the process seems more stable. However, since we increased the limit at least once, we got the same situation as we had with a lower memory limit.
Could you please tell is there any way to understand why the process consumes so much memory? Probably, we could make a memory dump of the process and provide it to you for future investigation?
Hi Anton,
I’m afraid there’s no other way but to dig into the situation and analyse the state and the data. Memory dump would be very helpful, indeed, but it’s not all we would need, all the SubGit logs are required, too, and also it would be great to have part of the SVN repository log that would include svn log for the revision SubGit goes OOM on, and, like, 100 revisions before and after this revision. If the OOME happens on a new Git commit translation, then it would be great to have similar part of Git log.
Hi Ildar.
Just to specify, do you mean a result of git log -100 and svn log -l 100 commands?
Hi Anton,
yes, for SVN and Git log I meant output of the log command, yet I’d also ask to add -v in the SVN log.
Hi Ildar.
The svn with a -v key contains sensitive information. Could you please tell me what exactly you need from the log? Probably, we could keep the information that you need and hide the rest that we don’t ready to show.
Hi Anton,
we need to know what exactly has been changed in every particular revision – that is, if some files were modified, or added, or a new branch created out of another branches, etc. We don’t need particular commit messages or even filenames.
Hi Ildar.
Sorry, let me clarify this. You need to know what exactly changed but you don’t need a diff?
We don’t create new branches, but I am not sure how to avoid showing you the diff, but show you the rest.
It would be helpful if you give me examples of what information are you expecting.
For example, for git, I could provide you with this:
commit e08413c66bb79e3faaee05140a78f939114f62f9
Author: Developer Name <dName@meraki.com>
xxxxxx commit message xxxxxxxx
Change-Id: Iaa09c9ea2e0b718e2a98409ae59d0eb775598fc1
Reviewed-on: https://gerrit.server.com/c/manage/+/191568
Reviewed-by: Another Developer <aDeveloper@meraki.com>
Style-Verified: Jenkins Build System <jenkins@meraki.com>
Tests-Verified: Jenkins Build System <jenkins@meraki.com>
for SVN:
Hi Anton,
sorry for being not clear enough.
We definitely do not need the actual changes, only need to understand what kind of change has been made in a new revision or commit – just some files modification, or some paths copying, etc., like this for SVN:
r5794 | root | 2019-05-31 20:42:20 +0500 (Пт, 31 май 2019) | 1 line
Changed paths:
A /pr4
A /pr4/branches
A /pr4/tags
A /pr4/trunkHello!
I’m working with Anton, and we were able to identify a few issues after performing several heap dumps. The main problem appears to be threads not terminating correctly. In the heap dump we took where subgit had exhausted its allocated 32GB of heap space, we saw 7378 threads in the waiting state . VisualVM showed them as "Piping consumer" daemon prio=5 tid=21411 TIMED_WAITING.
"Piping consumer" daemon prio=5 tid=23977 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.tmatesoft.svn.core.internal.io.svn.StreamLogger.lambda$new$0(StreamLogger.java:56)
local variable: org.tmatesoft.svn.core.internal.io.svn.StreamLogger#7342
local variable: java.util.logging.Logger#8
local variable: java.util.logging.Level#5
local variable: com.trilead.ssh2.channel.ChannelInputStream#14705
local variable: java.lang.String#744171
local variable: byte[]#124998
at org.tmatesoft.svn.core.internal.io.svn.StreamLogger$$Lambda$53.run(<unresolved string 0x0>)
at java.lang.Thread.run(Thread.java:750)
Opening the heap dump in Eclipse’s Memory Analyzer plugin totalled the occupied heap space from those threads at 28.8GB.
The stack trace of the stuck threads was identical to the problem identified in this SVNKit support topic: Replacing trilead-ssh2 with Apache sshd - #9 by joel-wenzel and I suspect that we’re seeing the same issue. If I understand the conversation in that topic, there was a fix pushed out with SVNKit 1.10.6. Our current version of Subgit (3.3.14) seems to be the latest, but it looks like it uses SVNKit 1.10.5:
[2022-06-02 10:49:42.999][subgit-install][1] SubGit version 3.3.14 ('Bobique') build #4433
[2022-06-02 10:49:43.000][subgit-install][1] System properties:
java.runtime.name : OpenJDK Runtime Environment
sun.boot.library.path : /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64
java.vm.version : 25.332-b09
java.vm.vendor : Oracle Corporation
java.vendor.url : http://java.oracle.com/
path.separator : :
java.vm.name : OpenJDK 64-Bit Server VM
file.encoding.pkg : sun.io
user.country : US
sun.java.launcher : SUN_STANDARD
sun.os.patch.level : unknown
jna.nosys : true
java.vm.specification.name : Java Virtual Machine Specification
user.dir : <redacted>
java.runtime.version : 1.8.0_332-8u332-ga-1~deb8u1-b09
java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment
java.endorsed.dirs : /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/endorsed
os.arch : amd64
java.io.tmpdir : /tmp
line.separator :
java.vm.specification.vendor : Oracle Corporation
os.name : Linux
sun.jnu.encoding : UTF-8
svnkit.http.methods : Digest,Basic,NTLM,Negotiate
java.library.path : /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-
gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
java.specification.name : Java Platform API Specification
java.class.version : 52.0
sun.management.compiler : HotSpot 64-Bit Tiered Compilers
svnkit.ssh.client : apache
sun.io.useCanonCaches : false
os.version : 4.9.0-0.bpo.6-amd64
user.home : <redacted>
user.timezone : America/Los_Angeles
java.awt.printerjob : sun.print.PSPrinterJob
file.encoding : UTF-8
java.specification.version : 1.8
java.class.path : /usr/share/java/subgit/jansi-1.6.jar:/usr/share/java/subgit/slf4j-nop-1.7.12.
jar:/usr/share/java/subgit/unirest-java-1.3.0.jar:/usr/share/java/subgit/javax.ws.rs-api-2.1.jar:/usr/share/j
ava/subgit/jetty-server-9.4.11.v20180605.jar:/usr/share/java/subgit/jetty-servlet-9.4.11.v20180605.jar:/usr/s
hare/java/subgit/jersey-server-2.27.jar:/usr/share/java/subgit/jersey-hk2-2.27.jar:/usr/share/java/subgit/jer
sey-container-jetty-servlet-2.27.jar:/usr/share/java/subgit/jersey-media-json-jackson-2.27.jar:/usr/share/jav
a/subgit/svnkit-1.10.5-snapshot20220408143050.jar:/usr/share/java/subgit/org.eclipse.jgit-5.0.11-sgk.jar:/usr
/share/java/subgit/annotations-7.0.3.jar:/usr/share/java/subgit/httpclient-4.3.jar:/usr/share/java/subgit/htt
pasyncclient-4.0-beta4.jar:/usr/share/java/subgit/httpmime-4.3.jar:/usr/share/java/subgit/json-20090211.jar:/
usr/share/java/subgit/javax.servlet-api-3.1.0.jar:/usr/share/java/subgit/jetty-http-9.4.11.v20180605.jar:/usr
/share/java/subgit/jetty-io-9.4.11.v20180605.jar:/usr/share/java/subgit/jetty-security-9.4.11.v20180605.jar:/
usr/share/java/subgit/jersey-common-2.27.jar:/usr/share/java/subgit/jersey-client-2.27.jar:/usr/share/java/su
bgit/jersey-media-jaxb-2.27.jar:/usr/share/java/subgit/javax.annotation-api-1.2.jar:/usr/share/java/subgit/ja
vax.inject-2.5.0-b42.jar:/usr/share/java/subgit/validation-api-1.1.0.Final.jar:/usr/share/java/subgit/hk2-loc
ator-2.5.0-b42.jar:/usr/share/java/subgit/jersey-container-servlet-2.27.jar:/usr/share/java/subgit/jersey-con
tainer-jetty-http-2.27.jar:/usr/share/java/subgit/jetty-webapp-9.4.7.v20170914.jar:/usr/share/java/subgit/jer
sey-entity-filtering-2.27.jar:/usr/share/java/subgit/jackson-annotations-2.8.10.jar:/usr/share/java/subgit/ja
ckson-databind-2.8.10.jar:/usr/share/java/subgit/jackson-module-jaxb-annotations-2.8.10.jar:/usr/share/java/s
ubgit/sshd-core-2.8.0.jar:/usr/share/java/subgit/sshd-common-2.8.0.jar:/usr/share/java/subgit/eddsa-0.3.0.jar
:/usr/share/java/subgit/sequence-library-1.0.4.jar:/usr/share/java/subgit/sqljet-1.1.14.jar:/usr/share/java/s
ubgit/jna-5.6.0.jar:/usr/share/java/subgit/jna-platform-5.6.0.jar:/usr/share/java/subgit/trilead-ssh2-1.0.0-b
uild222.jar:/usr/share/java/subgit/jsch.agentproxy.connector-factory-0.0.7.jar:/usr/share/java/subgit/jsch.ag
entproxy.svnkit-trilead-ssh2-0.0.7.jar:/usr/share/java/subgit/lz4-java-1.4.1.jar:/usr/share/java/subgit/jsch-
0.1.54.jar:/usr/share/java/subgit/JavaEWAH-1.1.6.jar:/usr/share/java/subgit/bcpg-jdk15on-1.62.jar:/usr/share/
java/subgit/bcprov-jdk15on-1.62.jar:/usr/share/java/subgit/bcpkix-jdk15on-1.62.jar:/usr/share/java/subgit/htt
pcore-4.3.jar:/usr/share/java/subgit/commons-logging-1.1.3.jar:/usr/share/java/subgit/commons-codec-1.6.jar:/
usr/share/java/subgit/httpcore-nio-4.3-beta2.jar:/usr/share/java/subgit/jetty-util-9.4.11.v20180605.jar:/usr/
share/java/subgit/osgi-resource-locator-1.0.1.jar:/usr/share/java/subgit/aopalliance-repackaged-2.5.0-b42.jar
:/usr/share/java/subgit/hk2-api-2.5.0-b42.jar:/usr/share/java/subgit/hk2-utils-2.5.0-b42.jar:/usr/share/java/
subgit/javassist-3.22.0-CR2.jar:/usr/share/java/subgit/jersey-container-servlet-core-2.27.jar:/usr/share/java
/subgit/jetty-continuation-9.4.7.v20170914.jar:/usr/share/java/subgit/jetty-xml-9.4.7.v20170914.jar:/usr/shar
e/java/subgit/jackson-core-2.8.10.jar:/usr/share/java/subgit/jcl-over-slf4j-1.7.32.jar:/usr/share/java/subgit
/antlr-runtime-3.4.jar:/usr/share/java/subgit/jsch.agentproxy.core-0.0.7.jar:/usr/share/java/subgit/jsch.agen
tproxy.usocket-jna-0.0.7.jar:/usr/share/java/subgit/jsch.agentproxy.usocket-nc-0.0.7.jar:/usr/share/java/subg
it/jsch.agentproxy.sshagent-0.0.7.jar:/usr/share/java/subgit/jsch.agentproxy.pageant-0.0.7.jar:/usr/share/jav
a/subgit/javax.inject-1.jar:/usr/share/java/subgit/slf4j-api-1.7.32.jar:/usr/share/java/subgit/translator-3.3
.14.jar:/usr/share/java/subgit/svngitkit-2.3.4-SNAPSHOT_877c759.jar
In case it’s hard to read, what I’m referring to is the /usr/share/jav a/subgit/svnkit-1.10.5-snapshot20220408143050.jar in the java.class.path
Would it be possible to get a version of Subgit with an SVNKit version bump to 1.10.6? Alternatively, is there a manual workaround that we can complete ourselves?
Thanks for the information. Indeed this problem seems to be the same as Joel Wenzel’s. As I can see from svnkit-1.10.5-snapshot20220408143050.jar filename, SVNKit version corresponds to r10827 (committed on Fri Apr 8 15:40:47 2022). But the fix to the Joel Wenzel’s problem was done at r10828. So SVNKit >=1.10.6 could be the solution. We’re planning to release a new SubGit version with the latest SVNKit (it will be 1.10.7) at the beginning of the next week.
If you don’t want, you can use the following trick:
- Backup
GIT_REPO/subgit/lib
directory.
- Copy all JAR files from SubGit distribution’s
lib/subdirectory to
GIT_REPO/subgit/lib
directory.
- Replace the SVNKit jar in
GIT_REPO/subgit/lib
with SVNKit 1.10.6
- Edit
GIT_REPO/subgit/lib/.files
file to enumerate there all JAR files and libjnidispatch.so file using <filename>=<SHA-1> format. Except the “fat jar” file (subgit-XXX_fat.jar) — delete or comment out that line. You can use the following one-liner to do that:
for file in *; do echo -n $file=; sha1sum $file | cut -f 1 -d ' '; done
- Delete the “fat jar” from
GIT_REPO/subgit/lib/
directory.
It’s a general trick to replace any JAR. The “fat jar” just consists of all those JARs unpacked and piled together to a single JAR file, so you can just replace it with many JAR files.
And .files file is used for consistency check: if some file’s SHA-1 doesn’t match its record in .files file, “subgit install” tries to “repair” the installation by replacing all the JARs with a “fat jar” it creates from its distribution. So it’s important to have correct SHA-1 hashes there. This list is also used to load upon start.
Anyway it would be easier just to wait for a new release, I hope it will fix the problem.