Gitlab merge request - Failed to map commit

Hi. When I create merge request on Gitlab and try to merge, the error occured.
The repository is used for 1 month and there was no issue before.
I tried to remove fast-post-receive and fast-pre-receive in custom_hooks directory.
And add export GIT_QUARANTINE_PATH=$GIT_OBJECT_DIRECTORY right above if [ -x "$C_HOOK" ] ; then line.
However the error still exist.

This is version information I use.

  • Gitlab version: 15.0.2
  • Subgit version: 3.3.14

I attach the logs inside git repo which have issue and config file.
Please let me know What log means…

config (10.0 KB)
logs.tar.gz (2.2 MB)

This is log when trying to merge on gitlab.
How can I resolve this issue?
Please help me.
Thanks.

[2022-07-18 11:30:01.242][pre-receive-hook][1] Initialized memory cache logger.
[2022-07-18 11:30:01.251][pre-receive-hook][1] SubGit version 3.3.14 (‘Bobique’) build #4433
[2022-07-18 11:30:01.251][pre-receive-hook][1] Posix: Non-JNA platform is chosen.
[2022-07-18 11:30:01.259][pre-receive-hook][1] Command name: pre-receive
[2022-07-18 11:30:01.259][pre-receive-hook][1] Command argument: /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git
[2022-07-18 11:30:01.364][pre-receive-hook][1] Posix: Non-JNA platform is chosen.
[2022-07-18 11:30:01.365][pre-receive-hook][1] Initialized file logger, level: INFO, logs directory is: ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/logs’.
[2022-07-18 11:30:01.365][pre-receive-hook][1] SubGit version 3.3.14 (‘Bobique’) build #4433
[2022-07-18 11:30:01.366][pre-receive-hook][1] Set temporary directory: /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/tmp
[2022-07-18 11:30:01.366][pre-receive-hook][1] Temporary directory ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/tmp’ is writable.
[2022-07-18 11:30:01.508][pre-receive-hook][1] Quarantine path is: ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/./objects/tmp_objdir-incoming-ZoBEjC’.
[2022-07-18 11:30:01.509][pre-receive-hook][1] Objects directory path is: ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/./objects/tmp_objdir-incoming-ZoBEjC’.
[2022-07-18 11:30:01.509][pre-receive-hook][1] Alternate objects directory path is: ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/./objects’.
[2022-07-18 11:30:01.514][pre-receive-hook][1] Obtaining file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’.
[2022-07-18 11:30:01.515][pre-receive-hook][1] Obtained file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’
[2022-07-18 11:30:01.515][pre-receive-hook][1] Daemon info pid=635490; port=35685; address=127:0:0:1; user=git from file ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.pid’
[2022-07-18 11:30:01.516][pre-receive-hook][1] Executing [kill, -0, 635490]; environmentVariables={HOME=/var/opt/gitlab};workingDirectory=/var/opt/gitlab
[2022-07-18 11:30:01.518][pre-receive-hook][1] ‘kill -0 635490’ exit code = 0
[2022-07-18 11:30:01.523][pre-receive-hook][1] Read ‘(version (3.3.14 4433 ))’.
[2022-07-18 11:30:01.524][pre-receive-hook][1] Daemon version is ‘3.3.14-4433’.
[2022-07-18 11:30:01.524][pre-receive-hook][1] About to release lock: Lock [file=/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock; memory=java.util.concurrent.locks.ReentrantLock@2d6a9952[Locked by thread main]; fs=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid]]
[2022-07-18 11:30:01.524][pre-receive-hook][1] Trying to release file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’.
[2022-07-18 11:30:01.525][pre-receive-hook][1] Released file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’
[2022-07-18 11:30:01.525][pre-receive-hook][1] Closed random access file ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’
[2022-07-18 11:30:01.527][pre-receive-hook][1] Sent ‘(pre-receive (/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git )100:d1646d90d6e8364c4d4f72cce793968308a0fed7 cc36b94fd4c6d1fe1c82df153fa8111bf3e7bc6e refs/heads/master
((GIT_ALTERNATE_OBJECT_DIRECTORIES /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/./objects )(GIT_OBJECT_DIRECTORY /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/./objects/tmp_objdir-incoming-ZoBEjC )(GIT_QUARANTINE_PATH /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/./objects/tmp_objdir-incoming-ZoBEjC )))’.
[2022-07-18 11:30:01.549][pre-receive-hook][1] Read ‘(message (39:Fetching revisions from SVN repository: ))’.
[2022-07-18 11:30:01.553][pre-receive-hook][1] Read ‘(message (12: up to date ))’.
[2022-07-18 11:30:01.554][pre-receive-hook][1] Read ‘(message (34:Sending commits to SVN repository: ))’.
[2022-07-18 11:30:01.554][pre-receive-hook][1] Read ‘(error (183:Failed to push some refs to Subversion repository.
Failed to map commit cc36b94fd4c6d1fe1c82df153fa8111bf3e7bc6e to SVN revision
Missing tree 017f6095fac6bd738548c838a45043d1626f6826
3908:org.tmatesoft.translator.util.f: Failed to push some refs to Subversion repository.
at org.tmatesoft.translator.util.f.c(SourceFile:109)
at org.tmatesoft.translator.daemon.E.a(SourceFile:88)
at org.tmatesoft.translator.daemon.C.c(SourceFile:98)
at org.tmatesoft.translator.daemon.g.a(SourceFile:335)
at org.tmatesoft.translator.daemon.g.a(SourceFile:45)
at org.tmatesoft.translator.daemon.g$1.run(SourceFile:306)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.tmatesoft.translator.util.f: Failed to map commit cc36b94fd4c6d1fe1c82df153fa8111bf3e7bc6e to SVN revision
at org.tmatesoft.translator.util.f.c(SourceFile:109)
at org.tmatesoft.translator.util.f.b(SourceFile:75)
at org.tmatesoft.translator.k.aj.a(SourceFile:2243)
at org.tmatesoft.translator.k.aj.a(SourceFile:2124)
at org.tmatesoft.translator.k.d.h.a(SourceFile:391)
at org.tmatesoft.translator.k.d.h.a(SourceFile:173)
at org.tmatesoft.translator.daemon.G.a(SourceFile:493)
at org.tmatesoft.translator.daemon.G.call(SourceFile:475)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.tmatesoft.translator.daemon.L.a(SourceFile:436)
at org.tmatesoft.translator.daemon.E.run(SourceFile:261)
Caused by: com.syntevo.svngitkit.core.b.i: Failed to map commit cc36b94fd4c6d1fe1c82df153fa8111bf3e7bc6e to SVN revision
at com.syntevo.svngitkit.core.b.i.a(SourceFile:29)
at org.tmatesoft.translator.j.b.a(SourceFile:71)
at org.tmatesoft.translator.j.F.a(SourceFile:366)
at com.syntevo.svngitkit.core.c.O.c(SourceFile:43)
at com.syntevo.svngitkit.core.c.O.b(SourceFile:36)
at org.tmatesoft.translator.k.aj.a(SourceFile:2227)
… 8 more
Caused by: java.io.IOException: Failed to map commit cc36b94fd4c6d1fe1c82df153fa8111bf3e7bc6e to SVN revision
at org.tmatesoft.translator.j.s.c(SourceFile:526)
at org.tmatesoft.translator.j.s.d(SourceFile:536)
at org.tmatesoft.translator.j.s.b(SourceFile:169)
at org.tmatesoft.translator.j.r.include(SourceFile:624)
at org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:108)
at org.eclipse.jgit.revwalk.TopoSortGenerator.(TopoSortGenerator.java:46)
at org.eclipse.jgit.revwalk.StartGenerator.next(StartGenerator.java:145)
at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:443)
at org.tmatesoft.translator.j.s.k(SourceFile:419)
at org.tmatesoft.translator.j.s.a(SourceFile:405)
at org.tmatesoft.translator.j.q.a(SourceFile:82)
at org.tmatesoft.translator.j.b.a(SourceFile:114)
at org.tmatesoft.translator.j.b.a(SourceFile:65)
… 12 more
Caused by: com.syntevo.svngitkit.core.b.i: Missing tree 017f6095fac6bd738548c838a45043d1626f6826
at com.syntevo.svngitkit.core.b.i.a(SourceFile:29)
at org.tmatesoft.translator.k.at.a(SourceFile:325)
at org.tmatesoft.translator.k.at.a(SourceFile:218)
at org.tmatesoft.translator.j.B.a(SourceFile:35)
at org.tmatesoft.translator.j.s.c(SourceFile:521)
… 24 more
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree 017f6095fac6bd738548c838a45043d1626f6826
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
at org.eclipse.jgit.treewalk.CanonicalTreeParser.reset(CanonicalTreeParser.java:191)
at org.eclipse.jgit.treewalk.CanonicalTreeParser.(CanonicalTreeParser.java:86)
at org.eclipse.jgit.notes.NoteParser.(NoteParser.java:71)
at org.eclipse.jgit.notes.NoteParser.parse(NoteParser.java:58)
at org.eclipse.jgit.notes.FanoutBucket$LazyNoteBucket.load(FanoutBucket.java:347)
at org.eclipse.jgit.notes.FanoutBucket$LazyNoteBucket.getNote(FanoutBucket.java:314)
at org.eclipse.jgit.notes.FanoutBucket.getNote(FanoutBucket.java:82)
at org.eclipse.jgit.notes.NoteMap.get(NoteMap.java:189)
at org.tmatesoft.translator.k.at.a(SourceFile:315)
… 27 more
))’.
[2022-07-18 11:30:01.557][pre-receive-hook][1] Failed to push some refs to Subversion repository.
Failed to map commit cc36b94fd4c6d1fe1c82df153fa8111bf3e7bc6e to SVN revision
Missing tree 017f6095fac6bd738548c838a45043d1626f6826

org.tmatesoft.translator.util.f: Failed to push some refs to Subversion repository.
Failed to map commit cc36b94fd4c6d1fe1c82df153fa8111bf3e7bc6e to SVN revision
Missing tree 017f6095fac6bd738548c838a45043d1626f6826

    at org.tmatesoft.translator.util.f.b(SourceFile:27)
    at org.tmatesoft.translator.daemon.j.a(SourceFile:256)
    at org.tmatesoft.translator.d.c.b(SourceFile:109)
    at org.tmatesoft.translator.d.a.b(SourceFile:68)
    at org.tmatesoft.translator.d.h.b(SourceFile:66)
    at org.tmatesoft.translator.d.c.c(SourceFile:63)
    at org.tmatesoft.translator.SubGitHook.d(SourceFile:98)
    at org.tmatesoft.translator.SubGitHook.a(SourceFile:66)
    at org.tmatesoft.translator.SubGitHook.a(SourceFile:24)
    at org.tmatesoft.translator.process.f.d(SourceFile:136)
    at org.tmatesoft.translator.process.f.c(SourceFile:106)
    at org.tmatesoft.translator.SubGitHook.main(SourceFile:38)

[2022-07-18 11:30:01.560][pre-receive-hook][1] Failed to execute pre-receive hook command
org.tmatesoft.translator.util.f: Failed to push some refs to Subversion repository.
Failed to map commit cc36b94fd4c6d1fe1c82df153fa8111bf3e7bc6e to SVN revision
Missing tree 017f6095fac6bd738548c838a45043d1626f6826

    at org.tmatesoft.translator.util.f.b(SourceFile:27)
    at org.tmatesoft.translator.daemon.j.a(SourceFile:256)
    at org.tmatesoft.translator.d.c.b(SourceFile:109)
    at org.tmatesoft.translator.d.a.b(SourceFile:68)
    at org.tmatesoft.translator.d.h.b(SourceFile:66)
    at org.tmatesoft.translator.d.c.c(SourceFile:63)
    at org.tmatesoft.translator.SubGitHook.d(SourceFile:98)
    at org.tmatesoft.translator.SubGitHook.a(SourceFile:66)
    at org.tmatesoft.translator.SubGitHook.a(SourceFile:24)
    at org.tmatesoft.translator.process.f.d(SourceFile:136)
    at org.tmatesoft.translator.process.f.c(SourceFile:106)
    at org.tmatesoft.translator.SubGitHook.main(SourceFile:38)

Hello,

sorry for the delay with the response, we have received the logs and we are investigating the issue, but it’s not yet clear enough what is the root cause and how can it be solved. Meanwhile, could you please advise on the following: we’ve found that the error occurred for the very first time on July 18, and latest previous successful push to the repository happened on July, 14th. Have any changes in configuration been applied in between? I mean, has anything been changed on SVN server and in SVN repository and same for GitLab and the repository? Server software upgrade, backup restoring, permissions change, anything similar?

Looks like the latest message has been lost due to server maintenance, here is the message:

Hi, Thank you for reply.

From July, 14th there is no changes in subgit configuration and SVN server either SVN repository too.
Also no changes in Gitlab and git repository. In Gitlab server, software version upgrade and backup restore, permission changes not happend.

It’s hard to find reason why this issue happened cause I can’t debug program…
If more information needed to investigate the issue, let me know.
Let me know if any updates…
Thank you.

Hello,

thank you for the clarification.

May I additionally ask you to provide some details about the setup, namely:

  • where is this GitLab installed, is that just a linux machine or a container?
  • if that is a container, where is it running – just on a server, in K8S, in some cloud container service? Where is SubGit installed, in the same container with GitLab or somewhere outside?
  • is that a standalone server or clustered solution? If clustered, how many nodes are there in cluster and where resides the repository storage?
  • is there any software that may access the repositories and handle files inside – like, antivirus software, e.g.?

Also, could you please advise how had the previous changes in Git been made, were those merge requests as well or just direct pushes? For example, how has the commit on July 14th been made?
Besides, may I ask you to double check if anything was changed on the server? There are some discrepancies in logs that make me to believe something might have been changed, maybe not in the repository itself, but on the server.

Additionally, could you please collect the following information from the repository:

  • git log -v output
  • output of the git for-each-ref command
  • output of git log refs/svn/map -p
  • git cat-file --batch-check --batch-all-objects command output

All the above commands are supposed to run in the repository directory.

Thank you in advance.

Hi. This is information what you mentioned.

  • where is this GitLab installed, is that just a linux machine or a container?
    Gitlab is installed as docker container. The version I use is gitlab-ce:15.0.2-ce.0

  • if that is a container, where is it running – just on a server, in K8S, in some cloud container service? Where is SubGit installed, in the same container with GitLab or somewhere outside?
    The container is run on a server. Os version is Ubuntu 20.04 LTS.

  • is that a standalone server or clustered solution? If clustered, how many nodes are there in cluster and where resides the repository storage?
    Standalone server.

  • is there any software that may access the repositories and handle files inside – like, antivirus software, e.g.?
    No. I don’t use such additional software on Gitlab container.

  • Also, could you please advise how had the previous changes in Git been made, were those merge requests as well or just direct pushes? For example, how has the commit on July 14th been made?
    We used both merge request and direct push to git repository. On July 14th, two merge request is approved.

  • Besides, may I ask you to double check if anything was changed on the server? There are some discrepancies in logs that make me to believe something might have been changed, maybe not in the repository itself, but on the server.
    There was no change including repository itself and either Gitlab server too.
    I just applied the patch in other issue pages.
    ( I tried to remove fast-post-receive and fast-pre-receive in custom_hooks directory.
    And add export GIT_QUARANTINE_PATH=$GIT_OBJECT_DIRECTORY right above if [ -x "$C_HOOK" ] ; then line.)
    Can you tell me more detail about discrepancies you mentioned?

  • Additionally, could you please collect the following information from the repository:
    (1) git log -v output
    I attached log as file named git_log_v.txt.

(2) output of the git for-each-ref command
seojikim@dev:~/work/ci_cd/git/bootrom$ git for-each-ref
d1646d90d6e8364c4d4f72cce793968308a0fed7 commit refs/heads/master
d1646d90d6e8364c4d4f72cce793968308a0fed7 commit refs/remotes/origin/HEAD
d1646d90d6e8364c4d4f72cce793968308a0fed7 commit refs/remotes/origin/master

(3) output of git log refs/svn/map -p
seojikim@dev:~/work/ci_cd/git/bootrom$ git log refs/svn/map -p
fatal: ambiguous argument ‘refs/svn/map’: unknown revision or path not in the working tree.
Use ‘–’ to separate paths from revisions, like this:
‘git […] – […]’

Should I add something to .git/config file?

(4) git cat-file --batch-check --batch-all-objects command output
I attached log as file named git_cat_file.txt.

Thank you.

git_cat_file.txt (1.2 MB)
git_log_v.txt (342.2 KB)

Hello,

thank you for the answers and files.

Could you please additionally clarify where is SubGit tool is installed and where it runs: is it embedded into the GitLab container or it runs somewhere else? How does it access GitLab’s filesystem?
Regarding the commands – it looks like I forgot to mention that all those command should be invoked not in a working copy, but in the repository directory on GitLab server, could you please run them there?

I can’y reply my answer to the post.

There is a log when I try to edit or reply in subgit support website.

Sorry, new users can only put 2 links in a post.

How can I write my answer on that page?

2022년 7월 21일 (목) 오후 7:48, Ildar Khusainov via TMate Support <discourse@tmatesoft.com>님이 작성:

Hello,

it looks like your message wasn’t not accepted due to link rules violation, could you please try to answer the post once again? Should work well now.

Hi.

  • Could you please additionally clarify where is SubGit tool is installed and where it runs: is it embedded into the GitLab container or it runs somewhere else? How does it access GitLab’s filesystem?
    SubGit tool is installed in Gitlab docker container.

  • Command output
    I attached log output again. The log output is from the repository directory on Gitlab server.
    git_cat_file.txt (1.6 MB)
    git_for_each_ref.txt (8.7 KB)
    git_log_refs_svn_map.txt (959.5 KB)
    git_log_v.txt (342.2 KB)

Thank you.

Hello,
so far we don’t have full inderstanding of why the problem happens but have some ideas about it.

What happens?
When you merge a merge request, a “git push” happens inside GitLab. A new commit is created (e.g., 5341d6bace0087fa4df7629d879e679cfd7b64ce). Then SubGit walks through all new commits (each time it’s just one commit: e.g. 5341d6bace0087fa4df7629d879e679cfd7b64ce) and tries to resolve SVN revision for them to distinguish between old and new commits (old commits have SVN revisions attached).

The SVN revisions are stored externally in Git notes format but in refs/svn/map reference. Only the latest commit under refs/svn/map is used and this commit has a tree looking like (level 1):

040000 tree 2c6f59d7ef2e97b4b89712a88460c67b9d6211c1    00
040000 tree bec321fcd00c34920a34e0b57d548c9375793482    01
040000 tree cf731982dfe7a9e09fd89250bd8225787079d256    02
040000 tree ce6bc1b304c15395edd5069b65990705f0dbcec6    03
040000 tree 4b543061ef143c974dbc9b845d1b8ffd1ac86bc5    04
...
040000 tree ec1ac8deaac4d7eef5b4c466d7c2097e59d3f341    fa
040000 tree f3d70df766e8b37233f8cc7434b04ecc06b5954c    fb
040000 tree 988203b784aa17cc24219633753fe388c0bf3a33    fc
040000 tree 77fff17f4d8a33117e411e410a4532786581247f    fd
040000 tree 207c4dbdf8903660bb9219ea7150ff647fab8fa2    fe
040000 tree c743ea28588bfb2f592310915165d8b219e724c2    ff

The paths are just 2 hexadecimal digits corresponding to the first 2 digits of commits and the tree points to another tree object of format (level 2)

100644 blob fa0346acf34120e7905f466013102a02eb6c2ed4    0b12238c3212e759f5b056ea1b1088f9160247
100644 blob efa25af5e5fb12eae4f8c5ad8c66878b6a1b8602    2400b60aedd7fe51069a9177c71a25ac1411da
100644 blob 6f8a2ac65c0501085d1447ff73fa38977996ec6b    2693ef30617d71f16b014d219acea0e2066f72
100644 blob 13f4060bfc686510e2aaadce1b62c2ccd8b08e9f    2c2061611ea8938f2ce8fc971c146acd35da48
100644 blob 9d14407b9d481e83b9b750563da3d6624b297383    3479f8c98a5eee70d6a528f2299d54277a1bfa
100644 blob 0ea40048b40f3fd4bb4c7b72868ec90324d73917    37adf5641e0e5c027e850a9be15fd94820d464
...

where the paths are the remaining 38 digits of the commit and the blobs contain content like
$ git show f17dbb7d

r5160 trunk/fw

i.e. contains the revision number and the name of the branch.

In your case SubGit tried to resolve SVN revision for newly pushed commit 5341d6bace0087fa4df7629d879e679cfd7b64ce. To do that it looked inside refs/svn/map tip commit to find a record corresponding to “53” (the first 2 digits of the commit SHA-1) – this corresponds to level 1:

...
040000 tree 540c504c8bd266e7d16948a050bf79849b8c5b81    53
...

Then it tried to load the tree 540c504c8bd266e7d16948a050bf79849b8c5b81 to find there the rest of the SHA-1 (41d6bace0087fa4df7629d879e679cfd7b64ce, i.e. 5341d6bace0087fa4df7629d879e679cfd7b64ce without “53”). And surprisingly SubGit cannot find this object 540c504c8bd266e7d16948a050bf79849b8c5b81 in the repository.

This is strange because in the output of “git cat-file --batch-check --batch-all-objects” command the object is present and all other commands succeed (they would fail if the repository were inconsistent). So the object is clearly present in your repository, at least according to Git command but SubGit just doesn’t see it.

Why this happens?
We don’t know for sure but we suspect this could be related to GitLab’s deduplication feature. Maybe the objects were moved away to another repositories (maybe so called “pool repositories”) and these repositories could be linked to this one through object/info/alternates files. One more argument in favor of this guess: this “53” directory (540c504c8bd266e7d16948a050bf79849b8c5b81 tree) could exist for a while without changes, and usually such objects are the best candidates to be archived, rather than recently added objects.

So I would ask you to check:

  1. If you have object/info/alternates file and what it contains.
  2. If you have 540c504c8bd266e7d16948a050bf79849b8c5b81 object on the filesystem, ie. objects/54/0c504c8bd266e7d16948a050bf79849b8c5b81 file other the filesystem inside the Git repository directory.
  3. If you don’t have it, check whether it is packed using
for file in $(find objects/pack -name '*.idx'); do git verify-pack -v "$file" | grep 540c504c8bd266e7d16948a050bf79849b8c5b81 && echo "$file"; done

one-liner from the Git repository directory. The script will print the name of the pack file index (.idx file) where the object resides. If it doesn’t find anything and the object is not on filesystem and you have alternates, it would make sense to look the object up in that alternate directory. We just want to find out where the object actually is to realize why SubGit doesn’t see it.

Once you find out where the object resides and whether alternate object directories are used, we could proceed to resolve the issue.

If you know something relevant about your GitLab settings related to deduplication or “pool repositories”. Or if you know that this repository was forked or you have some backup system that could enforce deduplication, please share your knowledge, this could be helpful as well.

  1. If you have object/info/alternates file and what it contains.
    root@10:/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git# cat objects/info/alternates
    …/…/…/…/…/@pools/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.git/objectsroot@10:/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e

  2. If you have 540c504c8bd266e7d16948a050bf79849b8c5b81 object on the filesystem, ie. objects/54/0c504c8bd266e7d16948a050bf79849b8c5b81 file other the filesystem inside the Git repository directory.
    root@10:/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git# ls objects/54/0c504c8bd266e7d16948a050bf79849b8c5b81
    ls: cannot access ‘objects/54/0c504c8bd266e7d16948a050bf79849b8c5b81’: No such file or directory

  3. If you don’t have it, check whether it is packed using
    root@10:/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git# for file in $(find objects/pack -name ‘*.idx’); do git verify-pack -v “$file” | grep 540c504c8bd266e7d16948a050bf79849b8c5b81 && echo “$file”; done
    root@10:/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git#

When find object inisde alternate directory,
root@10:/var/opt/gitlab/git-data/repositories/@pools/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.git# for file in $(find objects/pack -name ‘*.idx’); do git verify-pack -v “$file” | grep 540c504c8bd266e7d16948a050bf79849b8c5b81 && echo “$file”; done
540c504c8bd266e7d16948a050bf79849b8c5b81 tree 462 410 40801288
3c6c1dbe1c969e1ef062e62bc1d0407c6cbf1f13 tree 8 20 40944363 1 540c504c8bd266e7d16948a050bf79849b8c5b81
3ec7a11a36889dd9d697d8ce6b37d91059743c43 tree 8 20 40957502 1 540c504c8bd266e7d16948a050bf79849b8c5b81
objects/pack/pack-ad2348665b13e91ec3c8ab05fd60bfb6399b113b.idx

I don’t know gitlab’s deduplication feature well…
I understand as some object files are packed to another repository directory from Gitlab deduplication feature, so subgit couldn’t find object. Am I understanding correctly?
And the object that subgit is looking for looks like located at alternative directory of pool repository.
So How can I resolve this issue?

Thank you.

Hi, If any updates, Please let me know.
Thank you.

Hello!

Sorry for the delay, it took some time to analyse the whole situation with the development team.

I\m afraid though we haven’t found obvious issues, from what you’ve checked and sent it looks like everything is normal and should work well, so further investigation is needed. May I ask you to copy pack-ad2348665b13e91ec3c8ab05fd60bfb6399b113b.idx and pack-ad2348665b13e91ec3c8ab05fd60bfb6399b113b.pack files to the main repository in the directory where all the pack files reside and then check if that changes the situation?

Hi.
We have used svn repository only after the issue happened and didn’t stop subgit on Gitlab container.
The git repository on Gitlab seemed not update since 14 Jul, but after copy pack-ad2348665b13e91ec3c8ab05fd60bfb6399b113b.idx and pack-ad2348665b13e91ec3c8ab05fd60bfb6399b113b.pack to objects/pack directory of main repository, it seems subgit works.

root@10:/var/opt/gitlab/git-data/repositories# cp -a @pools/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.git/objects/pack/pack-ad2348665b13e91ec3c8ab05fd60bfb6399b113b.idx @hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/objects/pack/
root@10:/var/opt/gitlab/git-data/repositories# cp -a @pools/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.git/objects/pack/pack-ad2348665b13e91ec3c8ab05fd60bfb6399b113b.pack @hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/objects/pack/

After copying pack files on the git repository, svn revisions are generated as few commit on the repository.
And I create new merge request on the git repository for test and there’s no problem when trying to merge.
Log from subgit looks fine like below.

root@10:/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git# cat subgit/logs/pre-receive-hook.0.log

[2022-08-10 07:47:22.854][pre-receive-hook][1] Initialized memory cache logger.
[2022-08-10 07:47:22.863][pre-receive-hook][1] SubGit version 3.3.14 (‘Bobique’) build #4433
[2022-08-10 07:47:22.863][pre-receive-hook][1] Posix: Non-JNA platform is chosen.
[2022-08-10 07:47:22.870][pre-receive-hook][1] Command name: pre-receive
[2022-08-10 07:47:22.870][pre-receive-hook][1] Command argument: /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git
[2022-08-10 07:47:22.975][pre-receive-hook][1] Posix: Non-JNA platform is chosen.
[2022-08-10 07:47:22.976][pre-receive-hook][1] Initialized file logger, level: INFO, logs directory is: ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/logs’.
[2022-08-10 07:47:22.976][pre-receive-hook][1] SubGit version 3.3.14 (‘Bobique’) build #4433
[2022-08-10 07:47:22.977][pre-receive-hook][1] Set temporary directory: /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/tmp
[2022-08-10 07:47:22.977][pre-receive-hook][1] Temporary directory ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/tmp’ is writable.
[2022-08-10 07:47:23.122][pre-receive-hook][1] Quarantine path is: ‘/var/opt/gitlab/git-data/repositories/+gitaly/tmp/quarantine-848f536a0f1d6032-1747430802’.
[2022-08-10 07:47:23.122][pre-receive-hook][1] Objects directory path is: ‘/var/opt/gitlab/git-data/repositories/+gitaly/tmp/quarantine-848f536a0f1d6032-1747430802’.
[2022-08-10 07:47:23.123][pre-receive-hook][1] Alternate objects directory path is: ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/objects’.
[2022-08-10 07:47:23.128][pre-receive-hook][1] Obtaining file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’.
[2022-08-10 07:47:23.128][pre-receive-hook][1] Obtained file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’
[2022-08-10 07:47:23.129][pre-receive-hook][1] Daemon info pid=635490; port=35685; address=127:0:0:1; user=git from file ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.pid’
[2022-08-10 07:47:23.130][pre-receive-hook][1] Executing [kill, -0, 635490]; environmentVariables={HOME=/var/opt/gitlab};workingDirectory=/var/opt/gitlab
[2022-08-10 07:47:23.132][pre-receive-hook][1] ‘kill -0 635490’ exit code = 0
[2022-08-10 07:47:23.137][pre-receive-hook][1] Read ‘(version (3.3.14 4433 ))’.
[2022-08-10 07:47:23.138][pre-receive-hook][1] Daemon version is ‘3.3.14-4433’.
[2022-08-10 07:47:23.138][pre-receive-hook][1] About to release lock: Lock [file=/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock; memory=java.util.concurrent.locks.ReentrantLock@2d6a9952[Locked by thread main]; fs=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid]]
[2022-08-10 07:47:23.138][pre-receive-hook][1] Trying to release file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’.
[2022-08-10 07:47:23.139][pre-receive-hook][1] Released file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’
[2022-08-10 07:47:23.139][pre-receive-hook][1] Closed random access file ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’
[2022-08-10 07:47:23.140][pre-receive-hook][1] Sent ‘(pre-receive (/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git )100:20e27dee767409b455954b6933a4fbf1ccbf504f b4bbec35656efa7c3490e9ad3faecaa6ce68c706 refs/heads/master
((GIT_ALTERNATE_OBJECT_DIRECTORIES /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/objects )(GIT_OBJECT_DIRECTORY /var/opt/gitlab/git-data/repositories/+gitaly/tmp/quarantine-848f536a0f1d6032-1747430802 )(GIT_QUARANTINE_PATH /var/opt/gitlab/git-data/repositories/+gitaly/tmp/quarantine-848f536a0f1d6032-1747430802 )))’.
[2022-08-10 07:47:23.277][pre-receive-hook][1] Read ‘(message (39:Fetching revisions from SVN repository: ))’.
[2022-08-10 07:47:23.282][pre-receive-hook][1] Read ‘(message (12: up to date ))’.
[2022-08-10 07:47:23.282][pre-receive-hook][1] Read ‘(message (34:Sending commits to SVN repository: ))’.
[2022-08-10 07:47:23.283][pre-receive-hook][1] Read ‘(message (19: b4bbec3 => r5224 ))’.
[2022-08-10 07:47:23.283][pre-receive-hook][1] Read ‘(message (27:Sync completed successfully ))’.
[2022-08-10 07:47:23.283][pre-receive-hook][1] Read ‘(ok (pre-receive ))’.

root@10:/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git# cat subgit/logs/post-receive-hook.0.log

[2022-08-10 07:47:23.673][post-receive-hook][1] Initialized memory cache logger.
[2022-08-10 07:47:23.682][post-receive-hook][1] SubGit version 3.3.14 (‘Bobique’) build #4433
[2022-08-10 07:47:23.682][post-receive-hook][1] Posix: Non-JNA platform is chosen.
[2022-08-10 07:47:23.690][post-receive-hook][1] Command name: post-receive
[2022-08-10 07:47:23.690][post-receive-hook][1] Command argument: /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git
[2022-08-10 07:47:23.796][post-receive-hook][1] Posix: Non-JNA platform is chosen.
[2022-08-10 07:47:23.797][post-receive-hook][1] Initialized file logger, level: INFO, logs directory is: ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/logs’.
[2022-08-10 07:47:23.797][post-receive-hook][1] SubGit version 3.3.14 (‘Bobique’) build #4433
[2022-08-10 07:47:23.798][post-receive-hook][1] Set temporary directory: /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/tmp
[2022-08-10 07:47:23.798][post-receive-hook][1] Temporary directory ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/tmp’ is writable.
[2022-08-10 07:47:23.807][post-receive-hook][1] Obtaining file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’.
[2022-08-10 07:47:23.808][post-receive-hook][1] Obtained file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’
[2022-08-10 07:47:23.808][post-receive-hook][1] Daemon info pid=635490; port=35685; address=127:0:0:1; user=git from file ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.pid’
[2022-08-10 07:47:23.810][post-receive-hook][1] Executing [kill, -0, 635490]; environmentVariables={HOME=/var/opt/gitlab};workingDirectory=/var/opt/gitlab
[2022-08-10 07:47:23.817][post-receive-hook][1] ‘kill -0 635490’ exit code = 0
[2022-08-10 07:47:23.822][post-receive-hook][1] Read ‘(version (3.3.14 4433 ))’.
[2022-08-10 07:47:23.823][post-receive-hook][1] Daemon version is ‘3.3.14-4433’.
[2022-08-10 07:47:23.824][post-receive-hook][1] About to release lock: Lock [file=/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock; memory=java.util.concurrent.locks.ReentrantLock@14899482[Locked by thread main]; fs=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid]]
[2022-08-10 07:47:23.824][post-receive-hook][1] Trying to release file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’.
[2022-08-10 07:47:23.824][post-receive-hook][1] Released file lock on ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’
[2022-08-10 07:47:23.824][post-receive-hook][1] Closed random access file ‘/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git/subgit/daemon.lock’
[2022-08-10 07:47:23.825][post-receive-hook][1] Sent ‘(post-receive (/var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git /var/opt/gitlab/git-data/repositories/@hashed/21/0e/210e3b160c355818509425b9d9e9fd3ea2e287f2c43a13e5be8817140db0b9e6.git )0: )’.
[2022-08-10 07:47:23.827][post-receive-hook][1] Read ‘(ok (post-receive ))’.

Then how can I resolve this issue in the future? Do I need to copy pack files from alternatve pool repository to main repository all the times?
Or Is there a way to prevent this issue?

Thank you.

Hello!

Thank you a lot for the feedback, glad to know it is working now!

I’m afraid though that the root cause is not yet completely clear, so we will investigate it further with the development team, I’ll let you know as we come up with something.

Hi.
We need to synchronize svn and git repo for project as quick as possible.
So If any updates, let me know…

And I have a question about git hook flow when git push generated to git repository.
pre-receive hook generate svn revision number and git commit generated, then post-receive hook make commit to svn repository. Is it wright?
Can you describe more detail?

Thank you

Hello!

from what you mentioned in you previous message:

After copying pack files on the git repository, svn revisions are generated as few commit on the repository.
And I create new merge request on the git repository for test and there’s no problem when trying to merge.

and from the log part you’ve sent I got an impression that the mirror started working normally after the file had been copied and thus the SVN and Git repositories should have been synchronized already.

Regarding the hooks – actually, it is the pre-receive hook that is responsible for the synchronization, it starts the daemon (or connects to it if the daemon is already running), passes al the information and daemon performs the actual synchronization.

Hi.
Yes. After copying pack files to the repository, svn and git repositories have been synchronized well.
But there is possibility that the issue can be reproduced in future if git objects are packed and deduplication run.
So I want exact way to prevent this issue.
If you find root cause and find a way to prevent this issue, then please let me know.

Thank you.

Sure, we are working now to investigate the issue to the very end thus determining a way to prevent it from happening. I’ll definitely let you know as we find anything.