Pre-receive error on merge at gitlab

nglazov · October 20, 2021, 10:08am

Good afternoon.
We have installed a subgit on the repository with gitlab and today an error occurred that was not there before. It happens when you press “merge” in gitlab. If push to the master, then everything works out correctly.
The log shows that for some reason the pre-receive sends a link to a non-existent commit.
Each time the hash of the commit is different.
Can you tell me what can be done in this case?

[2021-10-20 14:53:09.221][daemon][1] Initialized memory cache logger.
[2021-10-20 14:53:09.223][daemon][1] SubGit version 3.3.10 ('Bobique') build #4368
[2021-10-20 14:53:09.223][daemon][1] Posix: Non-JNA platform is chosen.
[2021-10-20 14:53:09.229][daemon][1] Command name: pre-receive
[2021-10-20 14:53:09.229][daemon][1] Command argument: --svn
[2021-10-20 14:53:09.230][daemon][1] Command argument: /var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/.
[2021-10-20 14:53:09.230][daemon][1] Command argument: --limit
[2021-10-20 14:53:09.230][daemon][1] Command argument: 1634723599035
[2021-10-20 14:53:09.287][daemon][1] Posix: Non-JNA platform is chosen.
[2021-10-20 14:53:09.288][daemon][1] Initialized file logger, level: INFO, logs directory is: '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/logs'.
[2021-10-20 14:53:09.288][daemon][1] SubGit version 3.3.10 ('Bobique') build #4368
[2021-10-20 14:53:09.289][daemon][1] System properties:
java.runtime.name             : OpenJDK Runtime Environment
sun.boot.library.path         : /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64
java.vm.version               : 25.292-b10
java.vm.vendor                : Private Build
java.vendor.url               : http://java.oracle.com/
path.separator                : :
java.vm.name                  : OpenJDK 64-Bit Server VM
file.encoding.pkg             : sun.io
user.country                  : US
sun.java.launcher             : SUN_STANDARD
sun.os.patch.level            : unknown
jna.nosys                     : true
java.vm.specification.name    : Java Virtual Machine Specification
user.dir                      : /
java.runtime.version          : 1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
java.awt.graphicsenv          : sun.awt.X11GraphicsEnvironment
java.endorsed.dirs            : /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/endorsed
os.arch                       : amd64
java.io.tmpdir                : /tmp
line.separator                : 

java.vm.specification.vendor  : Oracle Corporation
os.name                       : Linux
sun.jnu.encoding              : ANSI_X3.4-1968
java.library.path             : /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
java.specification.name       : Java Platform API Specification
java.class.version            : 52.0
sun.management.compiler       : HotSpot 64-Bit Tiered Compilers
os.version                    : 4.15.0-134-generic
user.home                     : /var/opt/gitlab
user.timezone                 : GMT+05:00
java.awt.printerjob           : sun.print.PSPrinterJob
file.encoding                 : ANSI_X3.4-1968
java.specification.version    : 1.8
java.class.path               : /var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/./subgit/lib/subgit-3.3.10_4368_fat.jar
user.name                     : git
java.vm.specification.version : 1.8
sun.java.command              : org.tmatesoft.translator.SubGitDaemon pre-receive --svn /var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/. --limit 1634723599035
java.home                     : /usr/lib/jvm/java-8-openjdk-amd64/jre
sun.arch.data.model           : 64
user.language                 : en
java.specification.vendor     : Oracle Corporation
awt.toolkit                   : sun.awt.X11.XToolkit
java.vm.info                  : mixed mode
java.version                  : 1.8.0_292
java.ext.dirs                 : /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext
sun.boot.class.path           : /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes
java.awt.headless             : true
java.vendor                   : Private Build
file.separator                : /
java.vendor.url.bug           : http://bugreport.sun.com/bugreport/
sun.io.unicode.encoding       : UnicodeLittle
sun.cpu.endian                : little
sun.cpu.isalist               : 

[2021-10-20 14:53:09.289][daemon][1] Temporary directory '/tmp' is writable.
[2021-10-20 14:53:09.290][daemon][1] Launching daemon.
[2021-10-20 14:53:09.291][daemon][1] Server socket instance created.
[2021-10-20 14:53:09.293][daemon][1] List of network interfaces obtained.
[2021-10-20 14:53:09.293][daemon][1] Network interface: name:docker0 (docker0)
[2021-10-20 14:53:09.293][daemon][1] Processing address: /fe80:0:0:0:42:d4ff:fecb:7f7d%docker0
[2021-10-20 14:53:09.293][daemon][1] Processing address: /172.17.0.1
[2021-10-20 14:53:09.294][daemon][1] Network interface: name:eth0 (eth0)
[2021-10-20 14:53:09.294][daemon][1] Processing address: /fe80:0:0:0:2267:7cff:fedd:f998%eth0
[2021-10-20 14:53:09.294][daemon][1] Processing address: /192.168.104.253
[2021-10-20 14:53:09.294][daemon][1] Network interface: name:lo (lo)
[2021-10-20 14:53:09.294][daemon][1] Processing address: /0:0:0:0:0:0:0:1%lo
[2021-10-20 14:53:09.294][daemon][1] Processing address: /127.0.0.1
[2021-10-20 14:53:09.294][daemon][1] Local loopback address found: /127.0.0.1.
[2021-10-20 14:53:09.295][daemon][1] Trying to use a free port for address '127.0.0.1'.
[2021-10-20 14:53:09.295][daemon][1] Daemon socket opened for port: 44699.
[2021-10-20 14:53:09.296][daemon][1] Daemon process pid obtained: 677.
[2021-10-20 14:53:09.297][daemon][1] Daemon (pid=677; port=44699; address=127:0:0:1; user=git) at (/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit).
[2021-10-20 14:53:09.298][daemon][20] Daemon thread started
[2021-10-20 14:53:09.298][daemon][20] 'pid=677; port=44699; address=127:0:0:1; user=git' was written into temporary file '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/subgit.6520969605620346891.pid'
[2021-10-20 14:53:09.299][daemon][20] Pid file '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/daemon.pid' deleted.
[2021-10-20 14:53:09.328][daemon][20] '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/subgit.6520969605620346891.pid' renamed to '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/daemon.pid'
[2021-10-20 14:53:09.338][daemon][20] Command started, commands running: 1.
[2021-10-20 14:53:09.340][daemon][20] Sent '(version (3.3.10 4368 ))'.
[2021-10-20 14:53:09.341][daemon][20] Read '(pre-receive (/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/. /var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git )100:d33442239e28aae8f309cbd7f5cb1d42b95bb9c4 a3f13ff30ddc20b14635df9867a9f0d68ddc8ff3 refs/heads/master
 ((GIT_ALTERNATE_OBJECT_DIRECTORIES /var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/objects )(GIT_OBJECT_DIRECTORY /var/opt/gitlab/git-data/repositories/+gitaly/tmp/quarantine-8d8cfdd166be6ae5-104668850 )))'.
[2021-10-20 14:53:09.383][daemon][24] Git objects path from hook: '/var/opt/gitlab/git-data/repositories/+gitaly/tmp/quarantine-8d8cfdd166be6ae5-104668850'.
[2021-10-20 14:53:09.384][daemon][24] Alternate Git objects path from hook: '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/objects'.
[2021-10-20 14:53:12.414][daemon][24] Sent '(formatted-error (1744:UNRECOVERABLE ERROR:
	Missing unknown a3f13ff30ddc20b14635df9867a9f0d68ddc8ff3

CURRENT STATE:
	Git object a3f13ff30ddc20b14635df9867a9f0d68ddc8ff3 can't be accessed in the Git repository /var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git.

POSSIBLE REASONS:
	1) Git server is out of memory, available disk space,
	   processes count, or open files limits;
	2) When pushing through SSH, the system user on behalf of which.
	   SubGit daemon is running is not allowed to read files created by SSH user.
TO RECOVER:
	1) Make sure Git server has enough memory, disk resources AND high enough
	   open handler and processes count limits.
	2) When pushing through SSH, either
	   A) make sure the SSH user is the same user on behalf of which the SubGit is running
	        AND
	      is the same as the system user which is the owner of the Git repository; OR
	   B) set 'core.shared' option to 'true' in SubGit configuration file, run
	        $ subgit install "/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git"
	      to apply changes and make sure the owner of the Git repository and SSH user
	      belong to the same system group.
	      For details regarding sharing Git repository for a system group see:
	        https://git-scm.com/docs/git-config#Documentation/git-config.txt-coresharedRepository

	Once you have (1) and (2) fixed, retry push.

TO REPORT:
	1) Get error log from the server at
	   '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/logs'

	2) Report an issue at http://issues.tmatesoft.com/

THANK YOU!
 2325:org.tmatesoft.translator.util.i: UNRECOVERABLE ERROR:
	Missing unknown a3f13ff30ddc20b14635df9867a9f0d68ddc8ff3

CURRENT STATE:
	Git object a3f13ff30ddc20b14635df9867a9f0d68ddc8ff3 can't be accessed in the Git repository /var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git.

POSSIBLE REASONS:
	1) Git server is out of memory, available disk space,
	   processes count, or open files limits;
	2) When pushing through SSH, the system user on behalf of which.
	   SubGit daemon is running is not allowed to read files created by SSH user.
TO RECOVER:
	1) Make sure Git server has enough memory, disk resources AND high enough
	   open handler and processes count limits.
	2) When pushing through SSH, either
	   A) make sure the SSH user is the same user on behalf of which the SubGit is running
	        AND
	      is the same as the system user which is the owner of the Git repository; OR
	   B) set 'core.shared' option to 'true' in SubGit configuration file, run
	        $ subgit install "/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git"
	      to apply changes and make sure the owner of the Git repository and SSH user
	      belong to the same system group.
	      For details regarding sharing Git repository for a system group see:
	        https://git-scm.com/docs/git-config#Documentation/git-config.txt-coresharedRepository

	Once you have (1) and (2) fixed, retry push.

TO REPORT:
	1) Get error log from the server at
	   '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/logs'

	2) Report an issue at http://issues.tmatesoft.com/

THANK YOU!
	at org.tmatesoft.translator.daemon.C.a(SourceFile:182)
	at org.tmatesoft.translator.daemon.C.b(SourceFile:165)
	at org.tmatesoft.translator.daemon.C.c(SourceFile:88)
	at org.tmatesoft.translator.daemon.g.a(SourceFile:337)
	at org.tmatesoft.translator.daemon.g.a(SourceFile:45)
	at org.tmatesoft.translator.daemon.g$1.run(SourceFile:308)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
 ))'.
[2021-10-20 14:53:12.414][daemon][24] Command completed, commands running: 0.
[2021-10-20 14:53:12.415][daemon][24] Executing daemon shutdown request, the reason is UNKNOWN.
[2021-10-20 14:53:12.415][daemon][24] Trying to obtain file lock on '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/daemon.lock'.
[2021-10-20 14:53:12.416][daemon][24] Succeeded attempt to obtain file lock on '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/daemon.lock'
[2021-10-20 14:53:12.416][daemon][24] Thread pool shutdown scheduled.
[2021-10-20 14:53:12.417][daemon][24] Daemon info pid=677; port=44699; address=127:0:0:1; user=git from file '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/daemon.pid'
[2021-10-20 14:53:12.417][daemon][24] Pid file '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/daemon.pid' deleted.
[2021-10-20 14:53:12.417][daemon][24] Pid file deleted, daemon will be shut down NOW.
[2021-10-20 14:53:12.417][daemon][24] Server socket closed.
[2021-10-20 14:53:12.417][daemon][24] HTTP server shut down.
[2021-10-20 14:53:12.417][daemon][24] Trying to release file lock on '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/daemon.lock'.
[2021-10-20 14:53:12.417][daemon][20] Socket closed
java.net.SocketException: Socket closed
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
	at java.net.ServerSocket.implAccept(ServerSocket.java:560)
	at java.net.ServerSocket.accept(ServerSocket.java:528)
	at org.tmatesoft.translator.daemon.g.n(SourceFile:227)
	at org.tmatesoft.translator.daemon.g.run(SourceFile:192)
	at java.lang.Thread.run(Thread.java:748)
[2021-10-20 14:53:12.417][daemon][24] Released file lock on '/var/opt/gitlab/git-data/repositories/@hashed/7b/1a/7b1a278f5abe8e9da907fc9c29dfd432d60dc76e17b0fabab659d2a508bc65c4.git/subgit/daemon.lock'

ildar.khusainov · October 20, 2021, 2:33pm

Hello Nikolay,

thank you for reaching out to us on that matter!

It looks GitLab team has introduced some internal changes in recent versions that causes this issue. We are working to fix it permanently in the next SubGit version. Meanwhile, here is a workaround that allows overcoming this issue:

in the custom_hooks directory in an affected repository remove or rename both fast-pre-receive and fast-post-receive hooks

edit the pre-receive hook adding the following line:

export GIT_QUARANTINE_PATH=$GIT_OBJECT_DIRECTORY

right above this line:

 if [ -x "$C_HOOK" ] ; then

save the edited hook.

This change will allow SubGit to get all the needed information and thus will resolve the issue. I will let you know as we release the new version that does not need such a workaround to be made.

nglazov · October 21, 2021, 4:51am

Thanks, it helped. We are waiting for the new version.

nglazov · October 7, 2022, 5:53am

Hello.
Has this problem been fixed in recent releases?

ildar.khusainov · October 7, 2022, 6:43am

Hello Nikolay,

it looks that I didn’t informed you about the new version, my sincere apologies about that.
Yes, we released a version that contains fixes for that issue, I would recommend installing the latest version 3.3.16 that’s been release just a couple of days ago, it contains important fixes for other issues.

nglazov · October 7, 2022, 6:44am

Thank you, will be updated