Error connection is not available request timed out during SVN to GIT migration

[ildar.khusainov]

Hi Santosh,

just duplicating the email I’ve sent you. Email and this forum are two ways we are using to provide support yet this forum is preferred one, so I suggest continuing here.
Here is the email:

we are still investigating the issue, yet as I wrote in the forum thread, we haven’t managed to re-produce the issue locally, so the reason is not yet clear enough.

It would be much of a help for us to have Bitbucket and SVN Mirror logs (both global and affected repository specific) along with the setup details –

Bitbucket version and edition, SVN Mirror add-on installation path, Bitbucket cluster configuration (how many node it has and do they reside in the same data center),

and also how do you access the Bitbucket – do you have a VPN or maybe an SSH tunnel and of there are any network limitation, like some TCP ports closed?

This information will help us much to understand the overall situation and find out the cause.

Meanwhile, since you have a possibility to communicate with Bitbucket using REST API, we can proceed configuring and running mirrors through add-on’s REST API.

Here is the API documentation:

TMate SubGit: SVN Mirror Add-On for Bitbucket Server REST API
To import a project with gitgc as false in a specific bitbucket repo, as you mentioned, follow the chapter 2 of the article.

The gc setting can be set in the additional configuration options like this:

…

“config” : {

"svn.triggerGitGC" : false,
“svn.fetchInterval” : 60,
“translate.eols” : false|true,
“translate.ignores” : false|true
}

…

Please let me know if you have need any help with the REST requests.

[santhosh (Sitaram)]

hi Ildar,

All of a sudden the plugin started working again and we have started the import. I’m still unsure why/how it happened.

[ildar.khusainov]

Hi Santosh,

glad to know ti started to work.

It’s pretty strange, however, that it’s back without any actions or reconfigurations, it may worth to proceed with the investigation to find out the cause checking the data I mentioned earlier today. IT would also worth to check the network connection and environment as such an issue most probably is caused by network issues.

[santhosh (Sitaram)]

hi Ildar,

Sure. I have asked out bitbucket admin team to un-install/re-install the plugin and it started working. But we have lost 3 crucial days around this.

I will get back to you with my progress again and thanks for all the support.

[santhosh (Sitaram)]

Hi Ildar,

I get the following Input/Output error while importing.

Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: Input/output error

svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’

at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SourceFile:112)

at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SourceFile:96)

at com.a.a.a.d.C.a(SourceFile:1893)

at com.a.a.a.d.C.a(SourceFile:1046)

at com.a.a.a.d.C.a(SourceFile:782)

at com.a.a.a.d.C.b(SourceFile:700)

… 24 common frames omitted

Caused by: java.io.IOException: Input/output error

at java.io.FileOutputStream.close0(Native Method)

I tried to “Stop” importing and it has been showing “cancelling” for almost 24 hours now.

Can you please let me know on how to proceed on this?

Thanks

[santhosh (Sitaram)]

Hi Ildar,

Any update on this please? We have hard deadlines and I request your suggestion.

[semyon.vadishev]

Hello Santosh,

Ildar is on vacation these days, so I’ll do my best to fix the problem.

1. Input/Output error.

   This is a generic error that comes from the file system. Unfortunately, it’s hard to say what exactly caused the failure, could your server admin scan through the NFS error log and see any relevant errors there?

   Also, double-check that the disk space is sufficient on all the Bitbucket nodes you’re running. If you have debug logging enabled for both Bitbucket app and SVN Mirror add-on, I’d recommend disabling the debug logging, at least temporarily, since excessive logging may eat up some disk space.

2. Import cancellation is stuck.

   The file system error got the import process stuck, so the cancellation is unable to complete. Unfortunately, the only way out is restarting the plugin: go to Administration | Manage apps | SVN Mirror for Bitbucket Server, click on Disable button there, wait until the plugin is unloaded and then enable it back.

   Please let us know if restarting the plugin helps.

[santhosh (Sitaram)]

Hi Semyon,

Thank you for your reply.

I disabled the SVN Mirror for bitbucket server and enabled it back but I keep getting the following to check the status.

I checked the space and we have space on the nodes.

Can you please suggest the next actions.

[semyon.vadishev]

Hello Santosh,

Thank you for the update. I’ve tried analyzing the log files you’ve provided us earlier and I believe you’re facing two major problems with our add-on:

1. Sporadic file system errors.

   Looks like these errors occur during automatic mirror configuration and, as result, our add-on has to restart the process all over again. That leads to incomplete mirror configuration and the resulted branches mapping leads to inefficient import, e.g. fetching some tags as regular directories, which makes the import process extremely slow with enormous traffic between the SVN server and the Bitbucket node.

   I think there’s a pattern in those file system errors: usually, our add-on creates a temporary file and then immediately deletes or renames it. This happens when a config file gets generated during automatic mirror configuration and also when a new object gets written into the objects database. I saw both types of errors in the log files.

   There are two possible causes for such errors:

   • External process that interferes with short-lived files generated by our add-on. Please let me know if you’re using any kind of file scanners or any anti-virus software that theoretically may read those files in the background. It’s also possible that your operating system imposes some restrictions on the file system level that may interfere with our add-on, so let me know what OS you’re running. Finally, let me know if you’re running a Docker container for a Bitbucket node and if so, what image do you use for that, what version of Docker?

   • NFS-specific errors, e.g. network sync error on some short-lived files. Your server admin should be able to see those errors somewhere in the error log. You might also need to enable debug logging on the NFS level. Let me know if you can find anything there.

2. Timeout reading mirror data and similar errors.

   These are UI-specific errors caused by a network failure. Connection is getting lost somewhere between your browser and a Bitbucket node. Could you elaborate more on your current setup: what load balancer do you use for the Bitbucket cluster? Can you try accessing a specific Bitbucket note bypassing the balancer?

   You also mentioned that you’re using a VPN for accessing Bitbucket DC, is it possible to access a mirror settings page bypassing the VPN? Could you try accessing it from some other machine?

   Is the timeout error steadily reproducible for all the browsers on your machine?

[santhosh (Sitaram)]

Hi Semyon,

I checked on other OS and the same error persists. Also there is no scanner that scans files bypassing VPN. Docker container is not used.

Tried checking multiple browsers with private/incognito modes as well and the issue still persists.

It is not possible to bypass VPN and access bitbucket. I tried from other machines as well.

Yes. The timeout error is steadily reproducible for all the browsers on my machine and other machines as well.

Can you please suggest what else can be done?

[alexander.kitaev]

Hello Santhosh,

so far we are mostly out of ideas why this issue may happen - we had never encountered anything similar :(

What may give some clues is that “timeout” problem is not always reproducible - as far as I had understood it presented itself after you’ve restarted add-on. Also, a few months ago, that same issue resolved itself without any changes to the add-on or environment.

What I suppose might happen is that in your Data Center cluster of two nodes one SVN Mirror add-on doesn’t work properly on one of the nodes. If this would be the case, then balance loader may direct browser request to one node, and background request to another node (where add-on is not working or not replying properly).

To check this assumption I would suggest you to do the following, if that possible:

1. Temporary shutdown one of the Data Center cluster nodes, check that there is only one node displayed as running at Administration → Clustering page
2. Disable SVN Mirror Add-on: Administration → Manage Apps → SVN Mirror Add-on → Disable
3. Enable SVN Mirror Add-on: Administration → Manage Apps → SVN Mirror Add-on → Disable
4. Test if add-on UI works by going to Administration → Manage Apps → SVN Mirror Add-on → Configure page.

If problem persist, start the node you’ve shutdown back and disable another one. If my assumption is correct, this way we could find the node on which add-on is not working as expected and proceed from that.

In case issue persist even in single-node configuration then there might be some additional information in the browser dev console. To get that information:

1. press F12 when timeout error pop-up appears. Dev console will open
2. Navigate to Network tab
3. Press F5 or Ctrl-R to reload the page and collect network requests data
4. Click right mouse button for context menu and select “Save all as HAR” (or similar) - you will get the log file containing collected network requests, send it to support@subgit.com

With best regards,
Alexander Kitaev

[santhosh (Sitaram)]

Hi Alexander,

I have a quick question. Can we have an option to skip the current revision from migrating and proceed with the next one?
Please find the logs that show revisions “80381” through “80383” where the import stopped suddenly.

2021-08-01 09:37:33,762 sync - successfully received ‘branches/PECVD_XL_XT_3_1/devices/com/nvls/ncf/rt/device/mwhcontrol/simulation/implementation/SimMWHPowerControl.java’ with size=33234
2021-08-01 09:37:39,816 sync - fetched: hash = 957bfb95387c2a846ff1684dc57f5520c0398b78, branch = refs/svn/root/branches/PECVD_XL_XT_3_1, revision = 80382
2021-08-01 09:37:40,520 sync - Updating latest fetched revision for svn-remote “svn” to r80382
2021-08-01 09:37:40,809 sync - Getting content of “” directory at revision 80382
2021-08-01 09:37:40,851 sync - Getting content of “branches” directory at revision 80382
2021-08-01 09:37:43,471 sync - Getting content of “tags” directory at revision 80382
2021-08-01 09:38:24,979 sync - SET_PATH ‘’ 80383 not empty depth=infinity
2021-08-01 09:38:24,995 sync - SET_PATH ‘branches’ 80383 not empty depth=infinity
2021-08-01 09:38:25,053 sync - SET_PATH ‘’ 80382 not empty depth=infinity
2021-08-01 09:38:25,053 sync - Switch from http://xxxxx/Project_2543_Program_Derivation_B5 to http://xxxxxxxx/branches/GSPANN_ENHANCEMENTS_3
2021-08-01 17:52:35,386 sync - sendCopyFrom=false

r80383
A branches/GSPANN_ENHANCEMENTS_3 (from branches:80381)

2021-08-01 17:52:35,835 sync - svn: E175002: Input/output error
svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’
2021-08-01 17:52:35,929 sync - svn: E175002: Input/output error
svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’ com.a.a.a.b.i: svn: E175002: Input/output error
svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’

Is there a way that this revision can be skipped and then proceed to next revision and this specific revision can be copied manually?

I am in the process of bringing the node down and follow the steps as suggested by you.

[semyon.vadishev]

Hello Santhosh,

@santhosh

Is there a way that this revision can be skipped and then proceed to next revision and this specific revision can be copied manually?

Unfortunately, SVN Mirror can’t skip any specific revisions.

In this particular case, I recommend you skip the whole branch completely as it was copied from ^/branches/ directory. That means SubGit has to fetch the whole ^/branches/ directory in order to fetch ^/branches/GSPANN_ENHANCEMENTS_3. This is clearly the reason why your import gets stuck and then fails with I/O error.

You can skip the problematic branch with svn.excludeBranches option that you can specify along with the branches mapping:

[svn]
    trunk = ...
    ...
    excludeBranches = branches/GSPANN_ENHANCEMENTS_3

You can add this option and then proceed with the failed import.

[santhosh (Sitaram)]

Hi Semyon,

When you say we can skip this specific branch, can that be skipped only when we initiate the migration process? Or can it be skipped at this point now?
80383 revisions have already been migrated out of 81824 revisions at the moment.

If we can skip the branch at this point, can you provide the clear steps/process to do so?

Thank you

[ildar.khusainov]

Hi Santosh, Ildar’s here

I’m afraid SVN Mirror add-on does not allow to skip a branch that already exists both in SVN and in Git on the fly, the mapping configuration should be set to skip the branch before the actual migration (or repository rebuilding) process start.

[santhosh (Sitaram)]

Hi Ildar,

I keep getting this error again.

2021-08-20 00:08:28,337 sync - ######### ERROR REPORT ############
2021-08-20 00:08:28,393 sync - file ‘/bitbucket_home/shared/data/subgit/repositories/734/subgit/error’ does not exist
2021-08-20 00:08:28,420 sync - ######### GIT TO SVN FAILURE REPORT END ############
2021-08-20 00:08:28,468 sync - about to refresh repository
2021-08-20 00:08:28,992 sync - svn: E175002: Input/output error
svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’ org.tmatesoft.translator.util.f: svn: E175002: Input/output error
svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’
at org.tmatesoft.translator.util.f.c(SourceFile:109)
at org.tmatesoft.translator.util.f.b(SourceFile:75)
at org.tmatesoft.translator.m.aj.c(SourceFile:1000)

Found the above error about 4 hours ago.

But I currently see “Importing” status as below.

The cuurent logs:
Caused by: java.io.IOException: Input/output error
at java.io.FileOutputStream.close0(Native Method)
at java.io.FileOutputStream.access$000(FileOutputStream.java:53)
at java.io.FileOutputStream$1.close(FileOutputStream.java:356)
at java.io.FileDescriptor.closeAll(FileDescriptor.java:212)
at java.io.FileOutputStream.close(FileOutputStream.java:354)
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
at org.tmatesoft.svn.core.internal.io.dav.http.SpoolFile$SpoolOutputStream.close(SourceFile:247)
at org.tmatesoft.svn.core.internal.io.dav.http.SpoolFile$SpoolOutputStream.write(SourceFile:240)
at org.tmatesoft.svn.core.internal.wc.SVNCancellableOutputStream.write(SourceFile:44)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(SourceFile:814)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(SourceFile:848)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(SourceFile:220)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(SourceFile:451)
… 34 common frames omitted

2021-08-20 00:08:31,242 sync - ===
2021-08-20 00:09:33,312 sync - server 7.4.0 7004000; add-on 4.0.6; single node server
2021-08-20 00:09:34,795 sync - Repository fetch started.
2021-08-20 00:09:35,608 sync - svn authentication manager for ‘‘http://yyyyyyyy.com/svn/src’’
2021-08-20 00:09:35,633 sync - added user name credentials: xxyy
2021-08-20 00:09:35,639 sync - explicit user name credentials: xxyy
2021-08-20 00:09:35,724 sync - Checking remote ‘http://yyyyyyyy.com/svn/src’.
2021-08-20 00:09:37,293 sync - Revisions: <fetched=80382; last=81977>
2021-08-20 00:09:38,041 sync - svn authentication manager for ‘‘http://yyyyyyyy.com/svn/src’’
2021-08-20 00:09:38,048 sync - added user name credentials: xxyy
2021-08-20 00:09:38,056 sync - explicit user name credentials: xxyy
2021-08-20 01:30:23,735 sync - progress range: Fetching revisions from r80382 to r81977
2021-08-20 01:30:25,026 sync - svn authentication manager for ‘‘http://yyyyyyyy.com/svn/src’’
2021-08-20 01:30:25,056 sync - added user name credentials: xxyy
2021-08-20 01:30:25,062 sync - explicit user name credentials: xxyy
2021-08-20 02:48:04,825 sync - Starting fetch-back preventing procedure.
2021-08-20 02:48:05,789 sync - svn authentication manager for ‘‘http://yyyyyyyy.com/svn/src’’
2021-08-20 02:48:05,795 sync - added user name credentials: xxyy
2021-08-20 02:48:05,808 sync - explicit user name credentials: xxyy

I do not see any progress for the last couple of weeks and we have lost precious time performing the workarounds suggested by you/team.
Can you please help take this forward?

Thanks

[ildar.khusainov]

Hello Santosh,

this particular issue that the import stops at the revision 80383 most probably caused by the erroneous branch branches/GSPANN_ENHANCEMENTS_3 that should be excluded from the import as Semyon mentioned. Judging from that the add-on stops again on the r80383 the branch hasn’t been excluded and the add-on still tries to import it thus hitting the issue. I’m afraid there is no way to exclude a single revision from the import, as we discussed, so excluding the branch is the only way to resolve it. Could you advise have you added the exclusion setting in the mapping configuration?

[santhosh (Sitaram)]

Hi Ildar,

Can the exclusion of the branch be performed at this stage? i.e at 80382?
If so, can you provide the steps to do so?

[ildar.khusainov]

Hi Santosh,

if that branch was first created in the r80382 (which seems to be true) then yes, it can be applied on this stage, just add the setting in the mapping configuration and apply the mapping configuration change. If, however, this branch was present in the previous history, then such a change would require the repository to be rebuilt.

[santhosh (Sitaram)]

Hi Ildar,

I updated the mapping configuration to exclude the specific branch but the commit failed to push and it considered old configuration (with out exclude) and that failed too.

Please find the logs:

2021-08-20 10:20:00,452 apply - Executing [chmod, -R, g+rwX, /bitbucket_home/shared/data/subgit/repositories/734/subgit/tmp]; environmentVariables={};workingDirectory=/bitbucket_home
2021-08-20 10:55:41,393 apply - Created installation report:
ErrorReport[description=‘Push rejected: SubGit installation is in progress.
Pushes will be allowed as soon as SubGit installation is completed.’, warning=true, git enabled=false, svn enabled=true]
2021-08-20 10:55:41,468 apply - svn authentication manager for ‘‘http://xxyy.com/svn/src’’

2021-08-20 10:55:47,878 apply - Cannot commit write to /bitbucket_home/shared/data/repositories/734/config org.tmatesoft.translator.util.f: Cannot commit write to /bitbucket_home/shared/data/repositories/734/config
at org.tmatesoft.translator.util.f.c(SourceFile:109)
at org.tmatesoft.translator.util.f.b(SourceFile:75)

Caused by: com.a.a.a.b.i: Cannot commit write to /bitbucket_home/shared/data/repositories/734/config
at com.a.a.a.b.i.a(SourceFile:29)

Caused by: java.io.IOException: Cannot commit write to /bitbucket_home/shared/data/repositories/734/config
at org.eclipse.jgit.storage.file.FileBasedConfig.save(SourceFile:262)

2021-08-20 10:55:54,067 verify - Unrecoverable error: git repository ‘734’ has no subversion remote configuration. org.tmatesoft.translator.util.f: Unrecoverable error: git repository ‘734’ has no subversion remote configuration.
at org.tmatesoft.translator.util.f.b(SourceFile:27)

2021-08-20 10:55:58,460 sync - ############## GIT LOG #############
2021-08-20 11:31:40,455 sync - Failed to map commit 159a123c22213fe7eb8c695055823b4aeb404a20 to SVN revision
2021-08-20 11:31:40,586 sync - Metadata storage supports single remote, but the number of public remotes is 0 com.a.a.a.b.i: Metadata storage supports single remote, but the number of public remotes is 0
at org.tmatesoft.translator.m.at.f(SourceFile:152)

2021-08-20 18:27:31,931 sync - Unrecoverable error: git repository ‘734’ has no subversion remote configuration.
2021-08-20 18:27:31,944 sync - Unrecoverable error: git repository ‘734’ has no subversion remote configuration. org.tmatesoft.translator.util.f: Unrecoverable error: git repository ‘734’ has no subversion remote configuration.

The importing status was “cancelling” as soon as I added the exclude branch step in the mapping configuration, however the push was rejected later.