Hi Nikita,
thank you for the logs.
Regarding the ‘svn log -v’ output – may we have the output for the “5.1.115.128” branch?
Hi, is it will be okay if I’ll send it to you by email? (To avoid this file appearing on the forum)
And if it’s okay, could you give me this email address?
I just sent an email with the same subject as the title of this topic.
Hi Nikita,
yes, we have received the log, thank you for that.
I’d like to ask you about the rebuilding attempts you made: could you please advise what exactly happened when you tried to re-build the repository, especially in the case when you tried to rebuild it from the revision 140000? What was the exact command you used for that and what was the result and messages/errors along the way of the rebuild?
Hello.
I’ve already described what happened after I tried the rebuild from revision in the 7th reply. I think I’ll try one more time right now and will give you know what will happen.
BTW, after I’ll run the rebuild from the revision, what will happen to the revisions we skipped with the .metadata file? Will they still be missed or SubGit will try to restore them?
Hello Nikita,
sorry for asking that question too often, I just had an impression that the rebuild attempts you mentioned in the recent reply were not those you mentioned earlier.
If the rebuild starts from a revision earlier than the skipped one, than SubGit will re-download the revision during the rebuild thus rebuilding it. But that, actually, is what we would like to recommend doing in this case: skipping a revision in a running mirror is not recommended approach, especially to resolve the “Checksum mismatch” issues as that approach may cause other issues like this in the future. The most reliable way is to rebuild repository, at least from the r157964, or from earlier revision, like r140000 you mentioned before. Use the following command for the rebuild:
subgit install --rebuild-from-revision 157964 /path/to/git/repositoryYeah, it’s crystal clear. We do so just to monitor the behavior of the SubGit, we’re not going to implement such an approach in “prod”.
I’ve started rebuild like ~3h ago, it’s still in such condition:
subgit install --rebuild-from-revision 140000 ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git
SubGit version 3.3.10 ('Bobique') build #4368
About to shut down background translation process.
Shutdown request sent to background translation process (pid 1038996).
Background translation process (pid 1038996) has received shutdown request and will exit NOW.
SHUTDOWN SUCCESSFUL
I’ll keep you updating, but it’s the same thing as I tried earlier, except the fact, that then we don’t skip revisions
thank you Nikita, will be waiting to hear from you.
Hello,
First, is it okay that SubGit takes like few hours to actually start? I’m not sure what subgit is doing at this time(after the shutdown and before fetching new revisions), maybe you could advise?
And second, we’ve got such error few times:
INSTALLATION FAILED
error: svn: E175002: Processing REPORT request response failed: XML document structures must start and end within the same entity. (/svn/midas/!svn/vcc/default)
error: svn: E175002: REPORT request failed on '/svn/midas/!svn/vcc/default'
error: Unexpected error has occurred; please report along with the logs ('/var/opt/gitlab/git-data/repositories/@hashed/ef/2d/subgit-install-20210122-012249.zip')
error: to http://issues.tmatesoft.com/, thank you!
As I understand, it caused by a lost connection between the host with SubGit and SVN-host? Or there may be other causes?
Hello Nikita,
this error
error: svn: E175002: Processing REPORT request response failed: XML document structures must start and end within the same entity. (/svn/midas/!svn/vcc/default)
is being caused by that very reason – lost communication between SVN and Git, SubGit had not been able to read the data and the error arose.
As for the start time – it can be expected depending on the repository size and the history, SubGit indeed needs some time to prepare everything prior to start the rebuild.
Hello.
To give you an update:
We had such an error on the same revision a few times, so it doesn’t look like it’s just a network error. We’ve skipped one “problematic” branch-in-branch and the rebuild went further, but also began to fail on a specific revision with the same XML error.
Maybe some update from you? I think we give you all logs that you were needed, do you found something?
Hello Nikita,
it can be a network issue all the same despite it’s being caused by the same revision every time depending on changes in the particular revision and the web-server setting – in cases, for example, when the revision is big in size (meaning it changes much data) and the web server has timeout set. So when SubGit tries to download the changes, the web server just cuts the communication at some point and that causes the error; in fact, that is the most frequent cause of this type of error we were meeting with. But it’s not the only cause, faults in the SVN repository may also be the reason.
Taking into account that even working around one problematic revision does not resolve the issue and the same error appears on another revisions, I’m inclining to an assumption that this most probably related to the server settings. So to resolve that XML error my suggestion is to either revise the web server settings or to switch to another protocol – svn+ssh or file, if it’s possible.
Speaking about the workaround – may I ask you how did you skipped the one “problematic” branch-in-branch? Did you mean you removed some branch from the configuration or that was the same revision-skipping technic?
I’m afraid little that we could add to that we already discussed: we went through all the logs you shared, but we haven’t managed to find the moment when the issue started and found no information that would shed any light on the reason why the issue started to appear in the first place. We found many occurrences of the “Checksum mismatch” errors in logs, but it’s expected for this type of issue, it continues to appear in logs once it happened. Such an issue can be resolved (actually, it should be resolved this way as it’s the most reliable) by the rebuilding, either full or from a revision; it may be tricky, however, to choose right revision for the rebuild-from-revision, that’s why we always check all logs to find the right way for the rebuild.
So, for the initial “Checksum mismatch” the solution is to rebuild the repository, at least from revision 157964 or earlier. We’re getting problems during the rebuild, however, that most probably are caused by the web server settings. A possible way to resolve this issue is to either change the server settings or to use other protocol that is not prone to this type of problems.
Yeah, I also thought about web-server settings, and that this revision might be too big (that branch-in-branch was just the situation). Okay, maybe we’ll try to use such a workaround with web-server, thanks.
About skipping the branch - sorry, I actually meant excluding it via subgit config.
Hello.
We’ve tried both workarounds - increasing web-server’s timeouts and svn-ssh.
In the first case, no changes followed - the same error as it was, in the same revision.
With configured svn-ssh we’ve got the next error(also in one pretty big revision):
INSTALLATION FAILED
error: svn: E160016: Can't get entries of non-directory
Hello Nikita,
may I ask you what is the SVN version you use? This error looks very similar to one we found a couple of years ago which was SVN bug actually, not SubGit’s:
https://issues.tmatesoft.com/issue/SGT-1275
https://issues.apache.org/jira/browse/SVN-4791
The SVN team has resolved the issue, but in new versions (release after November 2018, those are 1.9.10, 1.10.4, 1.11.1), so it would worth to upgrade SVN in case if your version is affected.
Thanks! You might be right, we have svn, version 1.8.19 (r1800620)
We’ll consider updating.