Strange issue with GitLab, 0 commits yet they're there

lingfish · December 9, 2020, 7:43am

Hi, I’m trying SubGit out, and totally lost.

I’ve now synced three SVN projects. One shows the normal GitLab stats at the top, number of commits, file sizes, types of files etc. The other two (and more in testing), show 0 commits, and may or may not show any sizes.

Untitled

I’m doing a remote, where SG is installed on the GL server, and the SVN repos are remote (using a svn:// URL format). This is on Debian 10.

All the actual commits are there, once clicking into the GL commits part.

I’ve done the following:

Followed doco to the letter
All perms are fine
Using the correct (hashed) GL repo paths
Zero errors in any SubGit, GitLab or svnserve logs
Tried both install and import modes
All projects use the standard structure of branches, tags, trunk
I’ve literally diff’d the working and non-working configs, and the only differences (obviously) are SVN path, and strangely, Java path (yet I’ve tried both with no luck)
Both authors and passwd are configured fine
Standard CLI used as per your doco, as the git user, both with and without layout

SubGit version 3.3.10 (‘Bobique’) build #4368
gitlab-ee 13.6.1-ee.0
openjdk 11.0.9 2020-10-20
OpenJDK Runtime Environment (build 11.0.9+11-post-Debian-1deb10u1)
OpenJDK 64-Bit Server VM (build 11.0.9+11-post-Debian-1deb10u1, mixed mode, sharing)

I note that the last version of GL mentioned compatibility-wise is 13.1, could my newer version be breaking things? I doubt it, as one of the syncs works just fine!

Any help appreciated.

ildar.khusainov · December 9, 2020, 8:01am

Hello,

if all the commits are there in the repositories and no errors in SubGit logs, then it looks most like a GitLab UI issue, most probably UI cache. So it would worth try to clear it as the first step. Could you please try to run the following command on the GL server:

gitlab-rake cache:clear

This command should be invoked on behalf of root user, not git.
Also, could you advise have you tried to restart GitLab and does restart make any difference?

lingfish · December 9, 2020, 8:28am

Sorry, amongst the 1000 other things I’ve tried (and mentioned here), that was one of them ;) The rake job didn’t help, nor did housekeeping and other various options.

I’m fairly sure a reboot has been tried too.

I just tried one of the same projects, using git svn, and it’s worked fine. Very weird.

ildar.khusainov · December 9, 2020, 12:36pm

Ok, thanks for explanation.
It’s not an issue we aware of then, we will need to investigate it further, I’ll come back to you as we get something. And just to be sure – the repositories themselves are ok, correct? I mean, all the branches and commits are there, repositories can be cloned and they receive pushes, right?

lingfish · December 10, 2020, 12:31am

Hi, yep, all things look ok; I’m just concerned about what I cannot see, that may also be broken.

lingfish · December 12, 2020, 4:54am

Any further thoughts? Can I assist at all?

ildar.khusainov · December 12, 2020, 8:26am

Hi,

we managed to re-create the issue on our GitLab server, so it does not look we need much assistance with that, unless you have some ideas, of course :)
As we found, that information is being updated by GitLab’s post-receive hook, so the issue would be gone after the very first push to a mirrored repository, so it’s a working way to work the issue around. It does not seem to be applicable to imported repositories, however, so we are looking into it to find out more.

lingfish · December 12, 2020, 11:18pm

Thanks, good to know it’s reproducible!

So are you saying it has to be a git push, or a SVN commit will trigger the workaround too? Or either?

And after that, everything will behave as normal for all commits?

I found it still didn’t work in the case of import.

lingfish · December 12, 2020, 11:42pm

Ok, I just tested, and a SVN commit after install doesn’t trigger any change (of course the commit shows up in GL). Only when a GitLab commit (only tested via GL, not CLI), then things update.

This includes after the status stuff is updated via a git commit… further SVN commits don’t update the commit count etc.

lingfish · December 13, 2020, 12:26am

I’ve also just noticed this in logs, after a SVN commit:

   [2020-12-13 10:41:35.729][daemon][19] Location '/var/opt/gitlab/git-data/repositories/@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.git' fetch completed, fetch took 151 ms.
[2020-12-13 10:41:35.729][daemon][19] Executing [/var/opt/gitlab/git-data/repositories/@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.git/hooks/post-receive]; environmentVariables={SVN_LAST_FETCHED_REVISION=4537, GL_REPOSITORY=project-10, GIT_DIR=., GL_ID=user-2, USER=git, HOME=/var/opt/gitlab};workingDirectory=/var/opt/gitlab/git-data/repositories/@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.git
[2020-12-13 10:41:35.729][daemon][19] The executable '/var/opt/gitlab/git-data/repositories/@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.git/hooks/post-receive' does not exist.
com.syntevo.svngitkit.core.b.i: The executable '/var/opt/gitlab/git-data/repositories/@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.git/hooks/post-receive' does not exist.
        at com.syntevo.svngitkit.core.a.a(SourceFile:114)
        at com.syntevo.svngitkit.core.a.a(SourceFile:65)
        at org.tmatesoft.translator.util.r.b(SourceFile:365)
        at org.tmatesoft.translator.util.r.a(SourceFile:111)
        at org.tmatesoft.translator.k.d.h.b(SourceFile:311)
        at org.tmatesoft.translator.k.d.h.a(SourceFile:270)
        at org.tmatesoft.translator.k.d.h.a(SourceFile:264)
        at org.tmatesoft.translator.k.d.h.a(SourceFile:255)
        at org.tmatesoft.translator.k.d.h.a(SourceFile:148)
        at org.tmatesoft.translator.daemon.J.a(SourceFile:526)
        at org.tmatesoft.translator.daemon.J.b(SourceFile:524)
        at org.tmatesoft.translator.daemon.H.call(SourceFile:502)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at org.tmatesoft.translator.daemon.L.a(SourceFile:436)
        at org.tmatesoft.translator.daemon.E.run(SourceFile:261)
[2020-12-13 10:41:35.730][daemon][19] Triggered GitLab post-receive hook, user: 'user-2', no output

Doesn’t seem to be running the custom_hooks?! That’s what is configured in the config.

Also weird is that last line, referring to user-2…

I’ve only seen this today, on a newly-configured project.

ildar.khusainov · December 14, 2020, 12:40pm

Hi!

The exception is actually expected as there is no post-receive-hook in the ‘hooks’ directory indeed. SubGit keeps its hooks in the ‘custom_hooks’ directory, that’s right, but in GitLab environment it also tries to invoke GitLab’s hook, that is why the exception appears in logs.
SVN commits would not trigger the UI updates, indeed, bit pushes on the Git side would. We are investigating the situation o find a way to update the UI with any operations, I’ll be keeping you informed on the progress.

ildar.khusainov · January 21, 2021, 7:07am

Hello,

we have investigated the issue, but I’m afraid we haven’t found a reliable way to updated that information in GitLab UI after fetching changes from SVN. This information on the page is being updated by GitLab’s post-receive hook which is not triggered during fetch from SVN, that is the cause of the issue; but taking into account absence of reliable way to invoke the hook or update that information directly from SubGit, we decided to not to introduce any features that would do that. The main workaround is to push to a imported/mirrored repository; another possible workaround it to try to invoke the post-receive hook manually:

https://docs.gitlab.com/ee/development/internal_api.html#postreceive

adding this call in the user-post-receive hook, for example, so that it’s invoked on SVN fetch.