SubGit: Failed to push some refs to Subversion repository

Hello!

Thank you for your perfect tool!

We have large enough subversion repository and use SubGit in GitLab for duplex synchronization (svn-git-svn) .

git branch branch129 07eb5d9d
git checkout branch129
git push --set-upstream origin branch129 -- take several milliseconds

after this push I see svn/branches/branch129, this is ok

git fetch
git merge --no-ff master
git push origin -- take about 20 MINUTES!!!

the last push finished with the following error:

git.exe push --progress "origin" branch129:branch129
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 239 bytes | 239.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0)
remote: Fetching revisions from SVN repository:
remote:   up to date
remote: Sending commits to SVN repository:
remote:
remote: SubGit ERROR REPORT (SubGit version 3.3.8 ('Bobique') build #4337):
remote:
remote: You've received this message because SubGit (http://subgit.com/) is installed in your repository
remote: and an error that needs to be dealt with has occurred in SubGit translation engine.
remote:
remote: TEMPORARY ERROR:
remote:   Failed to push some refs to Subversion repository.
remote: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
remote: Process finished with non-zero exit code: 128
remote:
remote:
remote: CURRENT STATE:
remote:   Both Git and Subversion repository are open for pushes or commits.
remote:   Your commit was not committed, but you may retry it.
remote:
remote: TO RECOVER:
remote:     A) Address the problem if possible and then retry commit
remote:   OR
remote:     B) Run on the server
remote:        $ subgit uninstall /var/opt/gitlab/git-data/repositories/rolis/conterra.git/.
remote:
remote:   IMPORTANT: As soon as SubGit is uninstalled, both Git and Subversion repositories
remote:              will become open, but no synchronization will take place.
remote:
remote: TO REPORT:
remote:   Report an issue at http://issues.tmatesoft.com/
remote:   You may find logs on the server at '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/./subgit/logs'
remote:
remote: THANK YOU!
To http://gitlab.rlisystems.ru/rolis/conterra.git
! [remote rejected]       branch129 -> branch129 (pre-receive hook declined)
error: failed to push some refs to 'http://gitlab.rlisystems.ru/rolis/conterra.git'


git did not exit cleanly (exit code 1) (1329953 ms @ 02.05.2020 20:39:58)

When I merge trunk into such branch based on the same commit with subversion there is no problem.

Any hints?

Thank you for your time.

BR,
Anatoly Shirokov

Hello Anatoly,

such an error may appear for a number of reasons so we would need a little more details to find out what’s happening there. Could you please provide us SubGit logs from that repository? They reside in the subgit/logs directory in the repository.

Ildar, thank you for your time. Here is a full log from the pre-receive-hook.0.log file:

[2020-05-02 17:17:51.323][pre-receive] SubGit version 3.3.8 ('Bobique')
[2020-05-02 17:17:51.324][pre-receive] Command name: pre-receive.
[2020-05-02 17:17:51.324][pre-receive] Current user: git.
[2020-05-02 17:17:51.324][pre-receive] SVN/SubGit repository path is resolved to '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/.'
[2020-05-02 17:17:51.324][pre-receive] Git repository path is resolved to '/var/opt/gitlab/git-data/repositories/rolis/conterra.git'
[2020-05-02 17:17:51.324][pre-receive] Checking hook input, input size = 103
[2020-05-02 17:17:51.324][pre-receive] Hook input: 07eb5d9d43e58898f6aff88bfee28d854a24c27b 24794e72f33fb48e385acba6e8ec858122015659 refs/heads/branch129

[2020-05-02 17:17:51.324][pre-receive] The input doesn't contain forbidden references.
[2020-05-02 17:17:51.324][pre-receive] Loading repository options for path: '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/.'
[2020-05-02 17:17:51.324][pre-receive] Loaded config option daemon.pidFile = 'subgit/daemon.pid'
[2020-05-02 17:17:51.324][pre-receive] Loaded config option daemon.classpath = 'subgit/lib'
[2020-05-02 17:17:51.324][pre-receive] Loaded config option daemon.javaOptions = 'null'
[2020-05-02 17:17:51.324][pre-receive] Loaded config option core.logs = 'subgit/logs'
[2020-05-02 17:17:51.324][pre-receive] Loaded config option core.errorReport = 'null'
[2020-05-02 17:17:51.324][pre-receive] Loaded config option core.javaHome = 'null'
[2020-05-02 17:17:51.324][pre-receive] Loaded config option daemon.javaHome = 'null'
[2020-05-02 17:17:51.324][pre-receive] Loaded config option core.logLevel = 'null'
[2020-05-02 17:17:51.325][pre-receive] Checking Gitlab-specific environment variables.
[2020-05-02 17:17:51.325][pre-receive] Environment variable GL_ID = 'user-8'.
[2020-05-02 17:17:51.325][pre-receive] Environment variable GL_REPOSITORY = 'project-1'
[2020-05-02 17:17:51.325][pre-receive] Checking whether '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/.' is ASCII
[2020-05-02 17:17:51.325][pre-receive] Checking whether '/var/opt/gitlab/git-data/repositories/rolis/conterra.git' is ASCII
[2020-05-02 17:17:51.325][pre-receive] GIT_QUARANTINE_PATH environment variable: /var/opt/gitlab/git-data/repositories/rolis/conterra.git/./objects/incoming-i0JzJC
[2020-05-02 17:17:51.325][pre-receive] No error report file found.
[2020-05-02 17:17:51.325][pre-receive] Creating packet for sending to daemon, estimated size = 474.
[2020-05-02 17:17:51.325][pre-receive] Calculating classpath for repository: '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/.'.
[2020-05-02 17:17:51.325][pre-receive] Resolved classpath: '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/./subgit/lib/subgit-3.3.8_4337_fat.jar'.
[2020-05-02 17:17:51.325][pre-receive] Environment variable JAVA_HOME = '/usr/lib/jvm/java-8-openjdk-amd64/jre'
[2020-05-02 17:17:51.325][pre-receive] Obtaining file lock on '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/./subgit/daemon.pid'.
[2020-05-02 17:17:51.325][pre-receive] Obtained file lock on '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/./subgit/daemon.pid'.
[2020-05-02 17:17:51.325][pre-receive] Daemon info (4460,35350,127.0.0.1) from file '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/./subgit/daemon.pid'.
[2020-05-02 17:17:51.325][pre-receive] Checking whether daemon is running.
[2020-05-02 17:17:51.325][pre-receive] Daemon process is running, pid = 4460.
[2020-05-02 17:17:51.325][pre-receive] Daemon is running.
[2020-05-02 17:17:51.325][pre-receive] Trying to connect to daemon 127.0.0.1:35350.
[2020-05-02 17:17:51.326][pre-receive] Read '(version (3.3.8 4337 ))'.
[2020-05-02 17:17:51.326][pre-receive] Daemon version is '3.3.8-4337'.
[2020-05-02 17:17:51.326][pre-receive] Trying to release lock '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/./subgit/daemon.lock'.
[2020-05-02 17:17:51.326][pre-receive] Released lock '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/./subgit/daemon.lock'.
[2020-05-02 17:17:51.326][pre-receive] Sent '(pre-receive (/var/opt/gitlab/git-data/repositories/rolis/conterra.git/. /var/opt/gitlab/git-data/repositories/rolis/conterra.git )103:07eb5d9d43e58898f6aff88bfee28d854a24c27b 24794e72f33fb48e385acba6e8ec858122015659 refs/heads/branch129
 ((19:GIT_QUARANTINE_PATH /var/opt/gitlab/git-data/repositories/rolis/conterra.git/./objects/incoming-i0JzJC )))'.
[2020-05-02 17:17:52.395][pre-receive] Received '(message (39:Fetching revisions from SVN repository: ))'.
[2020-05-02 17:17:52.434][pre-receive] Received '(message (12:  up to date ))'.
[2020-05-02 17:17:52.434][pre-receive] Received '(message (34:Sending commits to SVN repository: ))'.
[2020-05-02 17:39:59.449][pre-receive] Received '(error (175:Failed to push some refs to Subversion repository.
com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
Process finished with non-zero exit code: 128
3423:org.tmatesoft.translator.util.f: Failed to push some refs to Subversion repository.
        at org.tmatesoft.translator.util.f.c(SourceFile:109)
        at org.tmatesoft.translator.daemon.E.a(SourceFile:88)
        at org.tmatesoft.translator.daemon.C.c(SourceFile:103)
        at org.tmatesoft.translator.daemon.g.a(SourceFile:337)
        at org.tmatesoft.translator.daemon.g.a(SourceFile:45)
        at org.tmatesoft.translator.daemon.g$1.run(SourceFile:308)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.tmatesoft.translator.util.f: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at org.tmatesoft.translator.util.f.c(SourceFile:109)
        at org.tmatesoft.translator.util.f.b(SourceFile:75)
        at org.tmatesoft.translator.k.aj.a(SourceFile:2233)
        at org.tmatesoft.translator.k.aj.a(SourceFile:2116)
        at org.tmatesoft.translator.k.d.h.a(SourceFile:391)
        at org.tmatesoft.translator.k.d.h.a(SourceFile:173)
        at org.tmatesoft.translator.daemon.G.a(SourceFile:493)
        at org.tmatesoft.translator.daemon.G.call(SourceFile:475)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.tmatesoft.translator.daemon.L.a(SourceFile:436)
        at org.tmatesoft.translator.daemon.E.run(SourceFile:261)
Caused by: com.syntevo.svngitkit.core.b.i: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at com.syntevo.svngitkit.core.b.i.a(SourceFile:29)
        at com.syntevo.svngitkit.core.internal.am.a(SourceFile:639)
        at com.syntevo.svngitkit.core.internal.ae.c(SourceFile:474)
        at com.syntevo.svngitkit.core.internal.i.b.s.a(SourceFile:567)
        at com.syntevo.svngitkit.core.internal.i.b.s.a(SourceFile:560)
        at com.syntevo.svngitkit.core.internal.i.b.s.a(SourceFile:291)
        at com.syntevo.svngitkit.core.internal.i.b.s.d(SourceFile:243)
        at com.syntevo.svngitkit.core.internal.i.d.a(SourceFile:173)
        at com.syntevo.svngitkit.core.internal.i.b.j(SourceFile:164)
        at com.syntevo.svngitkit.core.internal.i.h.b(SourceFile:293)
        at com.syntevo.svngitkit.core.internal.i.g.d(SourceFile:118)
        at com.syntevo.svngitkit.core.internal.i.c.b.d(SourceFile:31)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:1107)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:663)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:599)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:248)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:188)
        at com.syntevo.svngitkit.core.c.O.c(SourceFile:43)
        at org.tmatesoft.translator.j.b.b.a(SourceFile:382)
        at org.tmatesoft.translator.j.b.b.a(SourceFile:295)
        at org.tmatesoft.translator.j.b.b.a(SourceFile:179)
        at org.tmatesoft.translator.j.F.a(SourceFile:369)
        at com.syntevo.svngitkit.core.c.O.c(SourceFile:43)
        at com.syntevo.svngitkit.core.c.O.b(SourceFile:36)
        at org.tmatesoft.translator.k.aj.a(SourceFile:2217)
        ... 8 more
Caused by: java.io.IOException: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at com.syntevo.svngitkit.core.b.a(SourceFile:240)
        at com.syntevo.svngitkit.core.b.read(SourceFile:181)
        at org.eclipse.jgit.diff.RawText.isBinary(RawText.java:280)
        at com.syntevo.svngitkit.core.internal.am.a(SourceFile:636)
        ... 31 more
Caused by: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at com.syntevo.svngitkit.core.b.a(SourceFile:235)
        ... 34 more
 ))'.
[2020-05-02 17:39:59.449][pre-receive] No error report file found.
[2020-05-02 17:39:59.450][pre-receive] Failed to push some refs to Subversion repository.
com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
Process finished with non-zero exit code: 128

[2020-05-02 17:39:59.450][pre-receive]   at hook_execute (hook.c:288)
[2020-05-02 17:39:59.450][pre-receive]   at internal_pre_receive (pre-receive.c:225)
[2020-05-02 17:39:59.450][pre-receive]

Let me know that you need more details.

Hi Anatoly,

thank you for the part of log, but I’m afraid that is not all we need. May we have full logs from subgit/logs directory instead?

Ildar, here is a full log https://yadi.sk/d/4AwtdFK0ujV4QQ ~65MB

Thank you, Anatoly!

For some reason the log don’t contain the configuration, though, which we expected to find there. May I additionally ask you to send us SubGit configuration file from the repository?
Also, could you please describe your setup a little? Do I understand correctly is that a GitLab server and SubGit installed in a GitLab repository? If yes, what are GitLab version which Git version is installed on that server? May it happen that there are some file permissions or files lock issues on SVN side that prevent the changes to be written on SVN side?

Thank you in advance!

We have GitLab Community Edition [12.10.1] running within docker container with host volumes

GitLab            12.10.1 (e658772bd63)
GitLab Shell      12.2.0
GitLab Workhorsev 8.30.1
GitLab API        v4
Ruby              2.6.5p114
Rails             6.0.2
PostgreSQL        11.7

Here is a configuration:

# This is SubGit configuration file.
#
# SubGit is a tool for instant and transparent SVN/Git synchronization.
#
# Please visit http://www.subgit.com/ for more information on SubGit,
# new versions and online support.

# 
# General options
# 
[core]
	# When core.shared is set to 'true', all files will be made group-writable.
	#
	# You MUST set this option to 'true' when more than one system account is used
	# to work with this repository (e.g. Apache for Subversion and SSH for Git).
	# All these accounts MUST then share the same primary group.
	shared = false

	# Path to the directory where logs are stored.
	logs = subgit/logs

	# Path to the authors mapping file or authors mapping helper program.
	#
	# There could be one or more authorsFile options specified. Options are processed
	# from the last to the first one specifed, until mapping is obtained.
	#
	# Option value could be one of the following:
	#
	#        text file with explicit authors mapping in git-svn format,
	#        that is described at 'http://www.kernel.org/pub/software/scm/git/docs/git-svn.html'
	#     OR
	#        path to an executable authors mapping helper program and its optional arguments;
	#        authors mapping helper program is expected to read data from the standard input
	#        and to provide output to the standard output. Input/output format is:
	#
	#        for Git to Subversion mapping:
	#               INPUT:
	#                FULL AUTHOR NAME
	#                AUTHOR EMAIL
	#               OUTPUT:
	#                SUBVERSION USER NAME
	#
	#        for Subversion to Git mapping:
	#               INPUT:
	#                SUBVERSION USER NAME
	#               OUTPUT:
	#                FULL AUTHOR NAME
	#                AUTHOR EMAIL
	#
	#        Sample authors mapping script could be found in 'subgit/samples' directory.
	#
	# When no authorsFile option is specified, automatically generated mapping will be used.
	authorsFile = subgit/authors.txt

	# Domain to be used to generate Git committer email when authors mapping
	# is not defined or does not provide email address for a particular committer.
	defaultDomain = gitlab.rlisystems.ru

	# Encoding to be used to store paths in Git tree objects.
	pathEncoding = UTF-8

# 
# Subversion to Git mapping options
# 
[svn]
	# Subversion repository URL
	url = http://svnsrv:8080/svn/rolis/conterra

	# Options below (trunk, branches, tags, shelves) define correspondece between Subversion
	# directories and Git references. Depending on the actual Subversion project layout and whether
	# all or only some of the branches have to be mirrored, these options might need to be adjusted.
	#
	#  Generic mapping syntax is:
	#    <Subversion-Path-Pattern>:<Git-Reference-Pattern>
	#
	#  Subversion paths are relative to the URL defined by the svn.url option.
	#
	#  Wildcard mappings:
	#    Use wildcard branches and tags mappings to define correspondence between set of Subversion
	#    directories that represent branches or tags and corresponding Git references:
	#
	#    branches = branches/*:refs/heads/*
	#    branches = branches/*/*:refs/heads/*/*
	#    branches = branches/*/project:refs/heads/*
	#    branches = branches/feature_*_2015:refs/heads/features/*
	#
	#    Wildcard related mapping syntax rules:
	#
	#    - Each mapping could include zero or more '*' on both sides (sides are separated by ':') of the definition.
	#    - Each path or namespace segment should contain no more than one '*'.
	#    - Number of '*' on the left side of the mapping must be equal to the number of '*' on the right side of it.
	#
	#  Explicit mappings:
	#    Use explicit mapping for the Subversion directory that represents trunk and optionally for
	#    individual branches or tags:
	#
	#    trunk = trunk:refs/heads/master
	#    branches = branches/1.0.x:refs/heads/1.0.x
	#    tags = tags/1.0.1:refs/tags/1.0.1
	#
	#  Shelves:
	#    Shelves are special branches SubGit creates in Subversion repository to mirror Git anonymous
	#    branches to. Shelves mapping is optional, when unspecified anonymous Git branches will not
	#    be mirrored. shelves option defines location for those special branches:
	#
	#    shelves = shelves/*:refs/shelves/*
	#
	#  Excluding branches, tags and paths from translaiton:
	#     Mappings could be refined by specifying patterns that controls what branches and tags
	#     are excluded from the translation.
	#
	#     excludeBranches = SIMPLE_PATTERN
	#
	#     Value of excludeBranches option is simple path matching pattern, with zero or one '*' symbol.
	#     Branch path relative to the Subversion project URL is matched against the pattern specified.
	#
	#     To include or exclude particular files or directories within any of the branches or tags,
	#     use the following options:
	#
	#     excludePath = PATTERN
	#     includePath = PATTERN
	#
	#     Value of excludePath and includePath options is a pattern in format same to that used in .gitignore file.
	#     Only non-recursive patterns are supported by includePath option (pattern should start with '/').
	#     Paths being matched against the patterns are relative to the branch or tag path.
	#     See 'http://git-scm.com/docs/gitignore' for complete documentation on the pattern format.
	#
	#     When branches, tags or paths are excluded from the translation with the help of one of the above options,
	#     modifications to those branches or paths made in Subversion project would not be translated to Git,
	#     as well as changes in Git repository made to those locations would not appear in Subversion.
	#
	# SubGit supports mirroring of Subversion projects that have no branches. To configure mirror
	# for such project do the following:
	#
	#    a) Define no mapping (no trunk, branches, tags and shelves options), and
	#       make sure that svn.url option value refers to the project root folder
	#       in Subversion repository;
	#    b) optionally set trunk = :refs/heads/referenceName to override default Git branch name.
	#
	# Paths in mappings are case-sensitive, i.e. trunk and Trunk are considered to be
	# different paths.
	#
	# Once defines, mapping could be extended (e.g. new branches could be added), but changes that
	# leave existing branches unmapped are not accepted.
	#
	# Mapping suggested by default reflects recommended Subversion project layout as described
	# at 'http://svnbook.red-bean.com/en/1.7/svn.tour.importing.html#svn.tour.importing.layout'
	trunk = trunk:refs/heads/master
	branches = branches/*:refs/heads/*
	branches = IP_BCG:refs/heads/IP_BCG_1
	branches = IP_REPORTFARMTRACE:refs/heads/IP_REPORTFARMTRACE_1
	branches = trunk1:refs/heads/trunk1_1
	tags = tags/*:refs/tags/*
	shelves = shelves/*:refs/shelves/*
	excludeBranches = branches/IP_IOMWND

	# Interval between checks for new changes in Subversion repository in seconds.
	fetchInterval = 60

	# Subversion connection establishment timeout in seconds.
	connectTimeout = 300

	# Subversion connection read timeout in seconds.
	readTimeout = 600

	# enable HTTP requests spooling to prevent server-side timeout when request processing takes long time. 
	httpSpooling = true

	# When 'git.keepGitCommitTime' is set to 'true', SVN commits will have the same dates
	# as corresponding Git commits, even if this violates monotonic increase of commit dates from revision to revision.
	# As a side effect some features that rely on dates order like 'revision dates' may work incorrectly
	# (see 'http://svnbook.red-bean.com/en/1.7/svn.tour.revs.specifiers.html' for 'revision dates' feature explanation).
	#
	# When 'git.keepGitCommitTime' is set to false, SubGit will try to preserve Git commit dates while translation to SVN,
	# as it is possible, but it will guarantee SVN dates to increase monotonically from revision to revision.
	keepGitCommitTime = false

	# Comma-separated list of [auth] sections ids where Subversion credentials are defined.
	# Credentials will be used both for read-only and write operations, and it is recommended to
	# define credentials of an account that has full access to the Subversion repository.
	auth = default

# 
# Subversion authentication options.
# Valid options in this section are:
#
#    passwords: path to the file that contains user/password pairs;
#
#    credentialHelper: path to the credential helper program and its optional arguments;
#                      program is expected to be non-interactive (no prompt) and
#                      to use Git credential helper input/output format;
#                      refer to 'https://www.kernel.org/pub/software/scm/git/docs/git-credential.html#IOFMT' for more details;
#
#                      Sample credential helper script could be found in 'subgit/samples' directory.
#
#    userName: user name to use to log into Subversion repository;
#
#    password: password to use with the explicitly specified user name;
#
#    sshKeyFile: for svn+ssh protocol, SSH private key to use; OpenSSH format is supported;
#    sshKeyFilePassphrase: passphrase to use to read SSH private key;
#
#    sslClientCertFile: for SSL protocol, client SSL client certificate to use; PKSC#12 format is supported;
#    sslClientCertPassphrase: passphrase to use to read SSL client certificate;
#
#    subversionConfigurationDirectory: path to Subversion configuration directory or '@default@' to use
#                                      current user Subversion configuration and credentials cache.
# 
[auth "default"]
	# Passwords file path. Passwords file contains user name/password pairs that should be
	# used to log into Subversion repository.  
	passwords = subgit/passwd
	useDefaultSubversionConfigurationDirectory = false
	subversionConfigurationDirectory = /var/opt/gitlab/.subversion

# 
# Options specific to the background translation process.
# 
[daemon]
	# Pid file path.
	pidFile = subgit/daemon.pid

	# Timeout in seconds; background translation process exits
	# after being idle for the specified amount of seconds.
	#
	# 0 seconds timeout means that translation process exits immediately.
	#
	# Larger timeout values may improve translation performance by reducing
	# translation process startup overhead.
	idleTimeout = infinity

	# Explicit translation process classpath or path to the directory
	# that contains jars that have to be on the process classpath.
	classpath = subgit/lib

[hooks]
	directory = custom_hooks

Ildar, maybe it’s importante, such error is occurred when we remerge master into branch later and push such branch.

Let me know if you need more details.

Hi Anatoly,

yes, this is important, thank you for letting us know.
I’m afraid, though, we got not all the information we need, the hook log contains the exception, but the daemon log doesn’t contain records for the time the error occurred. Could it be possible to push the ‘branch129’ once again and then collect logs, especially latest pre-receive-hook and daemon logs?

Ildar, I have repeated my case and got clean log since 14:05 UTC. I started the new branch ‘branch130’, at 14:10 you can see start of push with merged master which took ~20 minutes and final error:

To http://gitlab.rlisystems.ru/rolis/conterra.git                                                                                                            
! [remote rejected]       branch130 -> branch130 (pre-receive hook declined)                                                                               error: failed to push some refs to 'http://gitlab.rlisystems.ru/rolis/conterra.git' 

logs.tar.gz (65.5 KB)

Pay attention on 1 minute delay between poritions in daemon.0.log:

[2020-05-04 16:27:36.690][daemon][22] "closeDir"
[2020-05-04 16:28:29.769][daemon][22] "openDir" "branches/branch130/appserver/services" 27214

Is it normal?

Thank you Anatoly, that’s very helpful.

This 1 minute delay between those entries in the log are actually expected, SubGit actually writes the data at that time, so nothing suspicious there.
The process indeed took about 20 minutes, but according to the logs, SubGit was busy performing all the needed checks and translating the data all that time, the logs does now reveal any possible issues. It’s not completely clear how old is that commit you check out at the first step and how much data is being merged, but from the log it looks there’s quite a lot of data and those 20 minutes are needed to send it to SVN. This time can be decreased, for example, by switching off eols translating:

[translate]
    eols = false

but otherwise it looks reasonable.

Judging from the logs, the error appears when SubGit tries to translate ‘misc/Plot’ directory. A possible cause of the error is either a missing blob in the repository or wrong Git work. This can be checked by trying to read a blob from the affected branch:

git ls-tree -r refs/heads/branch130 | grep misc/Plot

this command will show blobs in the Plot directory; try to read them with cat-file command, especially big one:

git cat-file blob SHA1

It would also worth to check the repository database consistency with fsck:

git fsck --full --strict

As a workaround try to set the path to the Git binary in the SubGit configuration file:

[core]
    …
    gitPath = <path to Git binary>

Ildar, thank you so much for your help!

I try to merge master which newer then the specified commit (07eb5d9d) on only ~250 commits (since 15 Apr). Such merge with subversion takes ~5-10 seconds without any conflicts.

I have gathered all output from

git ls-tree -r refs/heads/branch130

and try to read all objects with

git cat-file blob SHA1

All objects have been successfully read with the following script:

@echo off
call git ls-tree -r refs/heads/branch130 >todelete.txt
for /F "tokens=3" %%A in (todelete.txt) do (
	echo %%A 
	call git cat-file blob %%A > nul && echo %ERRORLEVEL%
)
del todelete.txt

on my origin clone. The same situation with refs/heads/master.

I have run git fsck --full --strict on my origin clone:

>git fsck --full --strict                                                                                                       
Checking object directories: 100% (256/256), done.                                                                                                          
Checking objects: 100% (362697/362697), done.                                                                                                               
Checking connectivity: 362719, done.  

Should I run the command ‘git fsck --full --strict’ on bare repository on gitlab host too?

I have added in the section ‘[core]’ the gitPath option

[core]
    ...
    # Encoding to be used to store paths in Git tree objects.
    pathEncoding = UTF-8

    gitPath = /opt/gitlab/embedded/bin
    ...

, but I cannot see one in the log:

[2020-05-04 19:43:46.441][pre-receive] Loading repository options for path: '/var/opt/gitlab/git-data/repositories/rolis/conterra.git/.'
[2020-05-04 19:43:46.441][pre-receive] Loaded config option daemon.pidFile = 'subgit/daemon.pid'
[2020-05-04 19:43:46.441][pre-receive] Loaded config option daemon.classpath = 'subgit/lib'
[2020-05-04 19:43:46.441][pre-receive] Loaded config option daemon.javaOptions = 'null'
[2020-05-04 19:43:46.441][pre-receive] Loaded config option core.logs = 'subgit/logs'
[2020-05-04 19:43:46.441][pre-receive] Loaded config option core.errorReport = 'null'
[2020-05-04 19:43:46.441][pre-receive] Loaded config option core.javaHome = 'null'
[2020-05-04 19:43:46.441][pre-receive] Loaded config option daemon.javaHome = 'null'
[2020-05-04 19:43:46.441][pre-receive] Loaded config option core.logLevel = 'null'

Should I restart server to apply changes in the subgit config?

It seems to me gitPath = /opt/gitlab/embedded/bin does not influence on this situation. I have still problem with misc/Plot:

[2020-05-04 20:44:33.704][daemon][20] "openDir" "branches/branch130/misc" 27214
[2020-05-04 20:44:33.704][daemon][20] "openDir" "branches/branch130/misc/Plot" 27214
[2020-05-04 20:44:33.704][daemon][20] "abortEdit"
[2020-05-04 20:44:33.743][daemon][20] ######### LOCATION STATE REPORT ############
[2020-05-04 20:44:33.743][daemon][20] com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
org.tmatesoft.translator.util.f: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128

It seems to me there is a problem with reading from subversion, but we have checked subversion repository.

Any hints?

Hi Anatoly,

It looks I forgot to mention how to apply new settings, my apologies for that. If some settings are changed in the configuration file, those changes should be applied by ‘install’ command:

subgit install <REPO>

Note, that not all the settings changes can be applied on the fly, some of them require full repository rebuilding. For example, the core.gitPath can be applied, but translate.eols requires the repository to be rebuild from the very beginning.

Thank you, Ildar!

Can I reproduce these lines via command line tools (git, svn?):

[2020-05-04 21:13:33.092][daemon][20] "closeDir"
[2020-05-04 21:13:38.992][daemon][20] "openDir" "branches/branch130/misc" 27214
[2020-05-04 21:13:38.993][daemon][20] "openDir" "branches/branch130/misc/Plot" 27214
[2020-05-04 21:13:38.993][daemon][20] "abortEdit"
[2020-05-04 21:13:39.028][daemon][20] ######### LOCATION STATE REPORT ############
[2020-05-04 21:13:39.028][daemon][20] com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
org.tmatesoft.translator.util.f: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at org.tmatesoft.translator.util.f.c(SourceFile:109)
        at org.tmatesoft.translator.util.f.b(SourceFile:75)
        at org.tmatesoft.translator.k.aj.a(SourceFile:2233)
        at org.tmatesoft.translator.k.aj.a(SourceFile:2116)
        at org.tmatesoft.translator.k.d.h.a(SourceFile:391)
        at org.tmatesoft.translator.k.d.h.a(SourceFile:173)
        at org.tmatesoft.translator.daemon.G.a(SourceFile:493)
        at org.tmatesoft.translator.daemon.G.call(SourceFile:475)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.tmatesoft.translator.daemon.L.a(SourceFile:436)
        at org.tmatesoft.translator.daemon.E.run(SourceFile:261)
Caused by: com.syntevo.svngitkit.core.b.i: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at com.syntevo.svngitkit.core.b.i.a(SourceFile:29)
        at com.syntevo.svngitkit.core.internal.am.a(SourceFile:639)
        at com.syntevo.svngitkit.core.internal.ae.c(SourceFile:474)
        at com.syntevo.svngitkit.core.internal.i.b.s.a(SourceFile:567)
        at com.syntevo.svngitkit.core.internal.i.b.s.a(SourceFile:560)
        at com.syntevo.svngitkit.core.internal.i.b.s.a(SourceFile:291)
        at com.syntevo.svngitkit.core.internal.i.b.s.d(SourceFile:243)
        at com.syntevo.svngitkit.core.internal.i.d.a(SourceFile:173)
        at com.syntevo.svngitkit.core.internal.i.b.j(SourceFile:164)
        at com.syntevo.svngitkit.core.internal.i.h.b(SourceFile:293)
        at com.syntevo.svngitkit.core.internal.i.g.d(SourceFile:118)
        at com.syntevo.svngitkit.core.internal.i.c.b.d(SourceFile:31)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:1107)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:663)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:599)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:248)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:188)
        at com.syntevo.svngitkit.core.c.O.c(SourceFile:43)
        at org.tmatesoft.translator.j.b.b.a(SourceFile:382)
        at org.tmatesoft.translator.j.b.b.a(SourceFile:295)
        at org.tmatesoft.translator.j.b.b.a(SourceFile:179)
        at org.tmatesoft.translator.j.F.a(SourceFile:369)
        at com.syntevo.svngitkit.core.c.O.c(SourceFile:43)
        at com.syntevo.svngitkit.core.c.O.b(SourceFile:36)
        at org.tmatesoft.translator.k.aj.a(SourceFile:2217)
        ... 8 more
Caused by: java.io.IOException: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at com.syntevo.svngitkit.core.b.a(SourceFile:240)
        at com.syntevo.svngitkit.core.b.read(SourceFile:181)

?

Especially for

[2020-05-04 21:13:38.993][daemon][20] "openDir" "branches/branch130/misc/Plot" 27214

What is hidden behind this line?

How to understand what is gone wrong in translation process:

[2020-05-04 21:13:39.028][daemon][20] com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
org.tmatesoft.translator.util.f: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at org.tmatesoft.translator.util.f.c(SourceFile:109)

?

How to know which file is not passed through translation?

Hi Anatoly,

could you please advise have you run ‘subgit install’ to apply the changes? The error appear even after that, tight?
And another question that I overlooked earlier: do I understand correctly you ran the Git commands on a Windows workstation? It was a workstation with the repository working copy on it, right? If so, I’m afraid it’s not what the are supposed for – we actually suspected that Git is not working correctly (or blob is missing) on the GitLab server, not in a working copy, sorry for not to be specific enough. Could you please try to do the same on the GitLab server then?

Hi Ildar,

I have installed new config without translation.eols with subgit install:

 subgit install /var/opt/gitlab/git-data/repositories/rolis/conterra.git
SubGit version 3.3.6 ('Bobique') build #4276

About to shut down background translation process.
Shutdown request sent to background translation process (pid 10668).
Background translation process (pid 10668) has received shutdown request and will exit NOW.

SHUTDOWN SUCCESSFUL

Translating Subversion revisions to Git commits...

    Subversion revisions translated: 27214.
    Total time: 3 seconds.

INSTALLATION SUCCESSFUL

Now I see in the log git:

[2020-05-05 11:23:32.982][daemon][21] Executing [/opt/gitlab/embedded/bin/git, --version]; environmentVariables={};workingDirectory=/var/opt/gitlab/git-data/repositories/rolis/conterra.git

With the gitPath option push works as expected and took ~13 minutes:

git push --progress origin                                                                                                     
Enumerating objects: 1, done.                                                                                                                               Counting objects: 100% (1/1), done.                                                                                                                         
Writing objects: 100% (1/1), 238 bytes | 238.00 KiB/s, done.                                                                                                
Total 1 (delta 0), reused 0 (delta 0)                                                                                                                       
remote: Fetching revisions from SVN repository:                                                                                                             
remote:   up to date                                                                                                                                        
remote: Sending commits to SVN repository:                                                                                                                  
remote:   2e666b5 => r27215 branches/branch130                                                                                                              
remote: Sync completed successfully                                                                                                                         
remote:                                                                                                                                                     
remote: To create a merge request for branch130, visit:                                                                                                     
remote:   http://gitlab.rlisystems.ru/rolis/conterra/-/merge_requests/new?
merge_request%5Bsource_branch%5D=branch130                                        remote:                                                                                                                                                     
To http://gitlab.rlisystems.ru/rolis/conterra.git                                                                                                              
07eb5d9d43..2e666b500b  branch130 -> branch130        

According to check: yep, I run all checkings on my windows host.

Also I have run git fsck with GitLab administrative panel:

This repository was last checked May 4, 2020 8:17pm. The check passed.

Also I have checked now my repository in command line on gitlab server:

for sha1 in $(git ls-tree -r refs/heads/branch130 | grep misc/Plot | awk '{print $3}'); do git cat-file blob $sha1; done > /dev/null

and

root@gitlab:/var/opt/gitlab/git-data/repositories/rolis/conterra.git# git fsck --full --strict
Checking object directories: 100% (256/256), done.
Checking objects: 100% (646293/646293), done.
Checking connectivity: 643525, done.
dangling commit 6f87a18b25b44e80164af0e215918dab0ef7e39c
dangling commit fac1061ecf2ca6c42f7396cfe4829f1bbfda0501
dangling commit 24794e72f33fb48e385acba6e8ec858122015659
dangling commit ee8e30cef4446fa70e89fe81b3c49c12b339631d
dangling commit c09a30e742c5dd0d96d48daf16e6d2e8961d72d5
dangling commit b4c110a1212f6055bb97d4e834f21e4041088fa8
dangling commit d1e5546328c43ccf2a9b457802f2e8a192b75aea
dangling commit c1c15df2e0b59e1c43421d5cbf0ebb3099e55611

It seems to me everything is ok excepted dangling commits.

If you teach me how to know translation problem file, I can send you one to check translation module.

Once again, thank you so much Ildar for your big help!!!

Hi Anatoly,

glad to hear it works!
The ‘fsck’ from the admin panel is also OK, but the command-line version should work well, too. It should be invoked right in the repository directory, maybe that was the problems with that?
As for the commands – their main purpose was to check if Git works well and check if a blob is missing. The output indicates that everything’s good from that point of view, indeed.
Honestly, I haven’t got what transaction problem file do you mean? SubGit only writes all the events to its logs, there’s no special files for problems.

Ildar, I am glad too. Thank you so much!!!

I mean the following stack trace where problem occurred in org.tmatesoft.translator.util:

[2020-05-04 21:13:38.992][daemon][20] "openDir" "branches/branch130/misc" 27214
[2020-05-04 21:13:38.993][daemon][20] "openDir" "branches/branch130/misc/Plot" 27214
[2020-05-04 21:13:38.993][daemon][20] "abortEdit"
[2020-05-04 21:13:39.028][daemon][20] ######### LOCATION STATE REPORT ############
[2020-05-04 21:13:39.028][daemon][20] com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
org.tmatesoft.translator.util.f: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at org.tmatesoft.translator.util.f.c(SourceFile:109)
        at org.tmatesoft.translator.util.f.b(SourceFile:75)
        at org.tmatesoft.translator.k.aj.a(SourceFile:2233)
        at org.tmatesoft.translator.k.aj.a(SourceFile:2116)
        at org.tmatesoft.translator.k.d.h.a(SourceFile:391)
        at org.tmatesoft.translator.k.d.h.a(SourceFile:173)
        at org.tmatesoft.translator.daemon.G.a(SourceFile:493)
        at org.tmatesoft.translator.daemon.G.call(SourceFile:475)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.tmatesoft.translator.daemon.L.a(SourceFile:436)
        at org.tmatesoft.translator.daemon.E.run(SourceFile:261)
Caused by: com.syntevo.svngitkit.core.b.i: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at com.syntevo.svngitkit.core.b.i.a(SourceFile:29)
        at com.syntevo.svngitkit.core.internal.am.a(SourceFile:639)
        at com.syntevo.svngitkit.core.internal.ae.c(SourceFile:474)
        at com.syntevo.svngitkit.core.internal.i.b.s.a(SourceFile:567)
        at com.syntevo.svngitkit.core.internal.i.b.s.a(SourceFile:560)
        at com.syntevo.svngitkit.core.internal.i.b.s.a(SourceFile:291)
        at com.syntevo.svngitkit.core.internal.i.b.s.d(SourceFile:243)
        at com.syntevo.svngitkit.core.internal.i.d.a(SourceFile:173)
        at com.syntevo.svngitkit.core.internal.i.b.j(SourceFile:164)
        at com.syntevo.svngitkit.core.internal.i.h.b(SourceFile:293)
        at com.syntevo.svngitkit.core.internal.i.g.d(SourceFile:118)
        at com.syntevo.svngitkit.core.internal.i.c.b.d(SourceFile:31)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:1107)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:663)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:599)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:248)
        at com.syntevo.svngitkit.core.c.l.a(SourceFile:188)
        at com.syntevo.svngitkit.core.c.O.c(SourceFile:43)
        at org.tmatesoft.translator.j.b.b.a(SourceFile:382)
        at org.tmatesoft.translator.j.b.b.a(SourceFile:295)
        at org.tmatesoft.translator.j.b.b.a(SourceFile:179)
        at org.tmatesoft.translator.j.F.a(SourceFile:369)
        at com.syntevo.svngitkit.core.c.O.c(SourceFile:43)
        at com.syntevo.svngitkit.core.c.O.b(SourceFile:36)
        at org.tmatesoft.translator.k.aj.a(SourceFile:2217)
        ... 8 more
Caused by: java.io.IOException: com.syntevo.svngitkit.core.b.j: Process finished with non-zero exit code: 128
        at com.syntevo.svngitkit.core.b.a(SourceFile:240)
        at com.syntevo.svngitkit.core.b.read(SourceFile:181)

but I could be wrong.

Hi Anatoly,

I’m afraid I still don’t follow) Do you mean how to find a file that causes an exception if one occurs? If so, I’m not sure I can teach you much about that, the reason is not always a file and it’s not always obvious from the exception what the cause is, most of the times a deeper investigation is needed to determine all the circumstances and find the reason. In this case, we suspected that Git was not able to translate some blob, either due to incorrect work or due to broken blob, but it turned out the cause was that JGit was unable to translate, while Git solved it, so it’s not a file in this case either.

Ildar, thank you for your explanation!

Indeed external git does not have problem with such blob, the problem occurs only with JGit.