Our mirror stopped working and now won't restart

We have had a mirror working for months.

This afternoon is stopped working and I have been trying to get it restarted.

We ran and install to restart things but got his issue:

-sh-4.2$ subgit install ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git
SubGit version 3.3.10 (‘Bobique’) build #4368

About to shut down background translation process.
Shutdown request sent to background translation process (pid 14294).
Background translation process (pid 14294) is busy, will retry in 1 seconds.
Shutdown request sent to background translation process (pid 14294).
Background translation process (pid 14294) is busy, will retry in 2 seconds.
Shutdown request sent to background translation process (pid 14294).
Background translation process (pid 14294) is busy, will retry in 3 seconds.
Shutdown request sent to background translation process (pid 14294).
Background translation process (pid 14294) is busy, will retry in 4 seconds.
Shutdown request sent to background translation process (pid 14294).
Background translation process (pid 14294) is busy, will retry in 5 seconds.
Shutdown request sent to background translation process (pid 14294).
Background translation process (pid 14294) is busy, will retry in 5 seconds.
Shutdown request sent to background translation process (pid 14294).
Background translation process (pid 14294) is busy, will retry in 5 seconds.
Shutdown request sent to background translation process (pid 14294).
Background translation process (pid 14294) is busy, will retry in 5 seconds.
Shutdown request sent to background translation process (pid 14294).
Timeout exceeded shutting down background translation process (pid 14294).
Will force shutdown.
Background translation process (pid 14294) has been KILLED.

SHUTDOWN SUCCESSFUL

Translating Subversion revisions to Git commits…

client_loop: send disconnect: Connection reset

Do you have any advice on how we can trouble shoot, please? It is causing us a serious issue.

Hi Steve,

the primary source of the diagnostics information is SubGit log, could you please share all the logs from that repository? The connection reset message looks suspicious, are there any issues communicating with SVN?

Hi,

There are no issues communicating with svn but we have found out some more information.

We believe we need to perform an install with a rebuild to correct the issue but we attempted the following command:

subgit install --rebuild-from-revision 147140 ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git

After adding the lines to our cofig:

excludeBranches = branches/nonsense
excludeTags = tags/MR5361.25869
excludeTags = tags/MR5361.25953
excludeTags = tags/MR6923.25890
excludeTags = tags/MR7612.25916
excludeTags = tags/MR7612.25919
excludeTags = tags/MR7612.25939
excludeTags = tags/MR7612.25950
excludeTags = tags/MR7796.25876
excludeTags = tags/MR7796.25901
excludeTags = tags/MR7796.25906
excludeTags = tags/MR7796.25947
excludeTags = tags/MR7847.25899
excludeTags = tags/MR8028.25872
excludeTags = tags/MR8028.25873
excludeTags = tags/MR8028.25874
excludeTags = tags/MR8028.25879
excludeTags = tags/MR8028.25892
excludeTags = tags/MR8028.25905
excludeTags = tags/MR8131.25884
excludeTags = tags/MR8155.25895
excludeTags = tags/MR8155.25897

However, we got the error message:

[2021-11-23 11:00:35.910][subgit-install][1] Could not update ‘refs/svn/root/branches/nonsense’ to dc44299384542eec85475bfc312726936753805d
org.tmatesoft.translator.util.f: Could not update ‘refs/svn/root/branches/nonsense’ to dc44299384542eec85475bfc312726936753805d
at org.tmatesoft.translator.util.f.c(SourceFile:109)
at org.tmatesoft.translator.util.f.b(SourceFile:75)
at org.tmatesoft.translator.k.aj.a(SourceFile:1268)
at org.tmatesoft.translator.k.d.g.d(SourceFile:95)
at org.tmatesoft.translator.a.N.b(SourceFile:55)
at org.tmatesoft.translator.a.t.c(SourceFile:23)
at org.tmatesoft.translator.b.a(SourceFile:193)
at org.tmatesoft.translator.b.a(SourceFile:144)
at org.tmatesoft.translator.b.a(SourceFile:54)
at org.tmatesoft.translator.process.f.d(SourceFile:136)
at org.tmatesoft.translator.process.f.c(SourceFile:106)
at org.tmatesoft.translator.b.a(SourceFile:79)
at org.tmatesoft.translator.b.a(SourceFile:70)
at org.tmatesoft.translator.SubGit.main(SourceFile:10)
Caused by: com.syntevo.svngitkit.core.b.i: Could not update ‘refs/svn/root/branches/nonsense’ to dc44299384542eec85475bfc312726936753805d
at com.syntevo.svngitkit.core.b.i.a(SourceFile:29)
at com.syntevo.svngitkit.core.c.ay.a(SourceFile:56)
at com.syntevo.svngitkit.core.c.ay.a(SourceFile:22)
at org.tmatesoft.translator.k.aj.a(SourceFile:1231)
… 11 more
Caused by: java.io.IOException: Could not update ‘refs/svn/root/branches/nonsense’ to dc44299384542eec85475bfc312726936753805d
at com.syntevo.svngitkit.core.c.ay.a(SourceFile:50)

The ‘nonsense’ branch has not been modified for several months and is at revision r101179 which is before the revision number we specified in our command line.

Can you spot anything that we have done wrong?

We have fixed it. It looks like we needed to put r147140 rather than 147140

We ran the above command, re-registerd the licence and started the mirror up but no one can commit anything.

The daemon seems to be going through everyone’s names a lot more than I remember it doing before. Has something gone out of sync?

daemon.0.log (9.6 MB)
pre-receive-hook.0.log (1.2 MB)

Hi Steve,

thank you for the logs.

I haven’t found much load on the names check – the daemon does this check from time to time, indeed, but it does not seem to take much time, the whole check takes 2 seconds and in fact runs in background. I see another issue there, however, it looks that SVN and Git sides are in conflict:

svn: E160024: resource out of date; try updating
svn: E175002: CHECKOUT of '/svn/tradinghub/!svn/ver/147631/trunk': 409 Conflict (https://alexandria.fsl.local:10443) org.tmatesoft.translator.util.q: svn: E160024: File or directory 'trunk' is out of date; try updating
svn: E160024: resource out of date; try updating
svn: E175002: CHECKOUT of '/svn/tradinghub/!svn/ver/147631/trunk': 409 Conflict (https://alexandria.fsl.local:10443)

If there is a conflict, then nobody will be able to commit, indeed, since SubGit tries to synchronize the mirror and fails to do so. It’s not completely clear how did they go in conflict, but the first step is definitely the conflict resolution.