Hello Santosh,
I understand that, but I’m afraid there is no other way to change the setting but the settings in the UI. Do I understand correctly you are accessing the Bitbucket Server over a VPN tunnel? Could it be possible to do that from the local network where the server resides (at least to change the setting)? Also, have you tried to use another web-browser?
hi Ildar,
I have tried using
Issue is still the same.
Is there a way we can change these settings from the server in any configuration file.
Why is this behaving this way suddenly ? It was working fine until Friday. Do you want to check any logs to understand this.
Does disable/enable (or) uninstall/install of the plugin will help ?
Did it happen with any customer before ?
As due to pandemic everybody is working remote , from what I understand everyone should be definately using company VPN right . I dont understand when you say local network . Do you mean should we be in the office location network which is hosting the bitbucket server ?
I’m kind of stuck and have 2 BIG repositories pending.
hi Ildar,
We are stuck and unable to use your plugin. Looks like a dead end for us. Can you help.
Hi Santosh,
sure, we were trying to find a way to switch the Long Polling off in your situation. A possible way to do that is to use SVN Mirror add-on’s REST API, here is the request that switched the setting off:
curl -u 'USER:PASSWORD' -H "Content-Type: application/json" -H "X-Atlassian-Token:no-check" --data '{"enabled" : false, "interval" : 60}' "https://BITBUCKET_SERVER_URL/rest/svn/1.0/config?name=longPollOptions"
Replace the USER, PASSWORD and BITBUCKET_SERVER_URL placeholders with correspondent user, password and the URL. If you don’t have cURL on your computer (which is usually true for Windows) use any other means to sent a POST request, for example, Postman software.
Also, I’d like to ask if it’s true that you have many web browser tabs or windows opened with SVN Mirror UI opened in those tabs? Such a situation may also disturb long polling operations and this may be the reason it does not work today while it worked last week.
hi Ildar,
No luck . (Masking url and passwords)
# curl -k -u 'abcd:abcd' -H "Content-Type: application/json" -H "X-Atlassian-Token:no-check" --data '{"enabled" : false, "interval" : 60}' "https://OUR_URL/rest/svn/1.0/config?name=longPollOptions"
#
After executing above CURL , this is what I see
FYI - Previous to executing this command
Hi Santosh,
could you please also add the -v cURL option, just to make sure the request went well:
curl -v -k -u…
What about the number of tabs, could it be the case you have too many of them opened? Also, could you please advise are Bitbucket Server pages available and work well and if only SVN Mirror add-on pages are affected?
Hi Ildar,
I have closed all other tabs and tried - no luck
Have asked my colleague from other location in India - same error for him .
Have asked my colleague from US - same error for him .
bitbucket works perfectly fine , only issue with this particular plugin page.
Here is the curl output with verbose option. (Masked sensitive infromation with **)
[root@]# curl -v -k -u 'USER:PASS' -H "Content-Type: application/json" -H "X-Atlassian-Token:no-check" --data '{"enabled" : false, "interval" : 60}' "https://url/rest/svn/1.0/config?name=longPollOptions"
* About to connect() to *** port 443 (#0)
* Trying **...
* Connected to *** port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=**,ST=CA,C=US
* start date: Jul 13 18:06:19 2020 GMT
* expire date: Jul 13 18:06:19 2023 GMT
* common name: d**
* issuer: **
* Server auth using Basic with user '**'
> POST /rest/svn/1.0/config?name=longPollOptions HTTP/1.1
> Authorization: Basic Q2hha2lTYTpDaGludHVANDM5
> User-Agent: curl/7.29.0
> Host: **
> Accept: */*
> Content-Type: application/json
> X-Atlassian-Token:no-check
> Content-Length: 36
>
* upload completely sent off: 36 out of 36 bytes
< HTTP/1.1 500
< Content-Type: application/json
< Vary: accept-encoding
< X-AREQUESTID: @66KSMYx586x240284x30
< X-AUSERID: 906
< X-AUSERNAME: **
< X-Content-Type-Options: nosniff
< Set-Cookie: ApplicationGatewayAffinity=e20670a2de0f64d827dc4697ec998195d0bf74ae97a76d4c4122e00ce9902ac7;Path=/;Domain=**
< Set-Cookie: ApplicationGatewayAffinityCORS=e20670a2de0f64d827dc4697ec998195d0bf74ae97a76d4c4122e00ce9902ac7;Path=/;Domain=**;SameSite=None;Secure
< Date: Mon, 12 Jul 2021 16:49:09 GMT
< Connection: close
< Content-Length: 0
<
* Closing connection 0
[root@]#hi Ildar,
Any update on this please ? This is what we still see.
We are kind of running out of schedule and this is important for us
Hi Santosh,
we are investigating this right now, I’ll be keeping you informed.
Meanwhile, we could use the REST API for the mirroring operations, too, could you please advise what the changes or operations do you need to perform in repositories?
Hi Santosh,
we continue investigating the issue, yet haven’t managed to re-produce it in our test environment. May I ask you to share Bitbucket and SVN Mirror logs with us to investigate? I’m afraid add-on’s Support ZIP feature is not much of a help currently, but it’s possible to collect it directly from the filesystem. Also, could you please advise what is your Bitbucket Server version and edition, how many node are in the cluster (if that is Datacenter cluster), and how did you install the add-on – have you installed it from Atlassian Marketplace or in some other way?
Hi Ildar,
We just want to import a project with gitgc as false in a specific bitbucket repo . This is what the REST API that we need for now.
Hi Santosh,
just duplicating the email I’ve sent you. Email and this forum are two ways we are using to provide support yet this forum is preferred one, so I suggest continuing here.
Here is the email:
we are still investigating the issue, yet as I wrote in the forum thread, we haven’t managed to re-produce the issue locally, so the reason is not yet clear enough.
It would be much of a help for us to have Bitbucket and SVN Mirror logs (both global and affected repository specific) along with the setup details –
Bitbucket version and edition, SVN Mirror add-on installation path, Bitbucket cluster configuration (how many node it has and do they reside in the same data center),
and also how do you access the Bitbucket – do you have a VPN or maybe an SSH tunnel and of there are any network limitation, like some TCP ports closed?
This information will help us much to understand the overall situation and find out the cause.
Meanwhile, since you have a possibility to communicate with Bitbucket using REST API, we can proceed configuring and running mirrors through add-on’s REST API.
Here is the API documentation:
TMate SubGit: SVN Mirror Add-On for Bitbucket Server REST API
To import a project with gitgc as false in a specific bitbucket repo, as you mentioned, follow the chapter 2 of the article.The gc setting can be set in the additional configuration options like this:
…
“config” : {
"svn.triggerGitGC" : false,
“svn.fetchInterval” : 60,
“translate.eols” : false|true,
“translate.ignores” : false|true
}…
Please let me know if you have need any help with the REST requests.
hi Ildar,
All of a sudden the plugin started working again and we have started the import. I’m still unsure why/how it happened.
Hi Santosh,
glad to know ti started to work.
It’s pretty strange, however, that it’s back without any actions or reconfigurations, it may worth to proceed with the investigation to find out the cause checking the data I mentioned earlier today. IT would also worth to check the network connection and environment as such an issue most probably is caused by network issues.
hi Ildar,
Sure. I have asked out bitbucket admin team to un-install/re-install the plugin and it started working. But we have lost 3 crucial days around this.
I will get back to you with my progress again and thanks for all the support.
Hi Ildar,
I get the following Input/Output error while importing.
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: Input/output error
svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SourceFile:112)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SourceFile:96)
at com.a.a.a.d.C.a(SourceFile:1893)
at com.a.a.a.d.C.a(SourceFile:1046)
at com.a.a.a.d.C.a(SourceFile:782)
at com.a.a.a.d.C.b(SourceFile:700)
… 24 common frames omitted
Caused by: java.io.IOException: Input/output error
at java.io.FileOutputStream.close0(Native Method)
I tried to “Stop” importing and it has been showing “cancelling” for almost 24 hours now.
Can you please let me know on how to proceed on this?
Thanks
Hi Ildar,
Any update on this please? We have hard deadlines and I request your suggestion.
Hello Santosh,
Ildar is on vacation these days, so I’ll do my best to fix the problem.
Input/Output error.
This is a generic error that comes from the file system. Unfortunately, it’s hard to say what exactly caused the failure, could your server admin scan through the NFS error log and see any relevant errors there?
Also, double-check that the disk space is sufficient on all the Bitbucket nodes you’re running. If you have debug logging enabled for both Bitbucket app and SVN Mirror add-on, I’d recommend disabling the debug logging, at least temporarily, since excessive logging may eat up some disk space.
Import cancellation is stuck.
The file system error got the import process stuck, so the cancellation is unable to complete. Unfortunately, the only way out is restarting the plugin: go to Administration | Manage apps | SVN Mirror for Bitbucket Server, click on Disable button there, wait until the plugin is unloaded and then enable it back.
Please let us know if restarting the plugin helps.
Hi Semyon,
Thank you for your reply.
I disabled the SVN Mirror for bitbucket server and enabled it back but I keep getting the following to check the status.
I checked the space and we have space on the nodes.
Can you please suggest the next actions.
Hello Santosh,
Thank you for the update. I’ve tried analyzing the log files you’ve provided us earlier and I believe you’re facing two major problems with our add-on:
Sporadic file system errors.
Looks like these errors occur during automatic mirror configuration and, as result, our add-on has to restart the process all over again. That leads to incomplete mirror configuration and the resulted branches mapping leads to inefficient import, e.g. fetching some tags as regular directories, which makes the import process extremely slow with enormous traffic between the SVN server and the Bitbucket node.
I think there’s a pattern in those file system errors: usually, our add-on creates a temporary file and then immediately deletes or renames it. This happens when a config file gets generated during automatic mirror configuration and also when a new object gets written into the objects database. I saw both types of errors in the log files.
There are two possible causes for such errors:
External process that interferes with short-lived files generated by our add-on. Please let me know if you’re using any kind of file scanners or any anti-virus software that theoretically may read those files in the background. It’s also possible that your operating system imposes some restrictions on the file system level that may interfere with our add-on, so let me know what OS you’re running. Finally, let me know if you’re running a Docker container for a Bitbucket node and if so, what image do you use for that, what version of Docker?
NFS-specific errors, e.g. network sync error on some short-lived files. Your server admin should be able to see those errors somewhere in the error log. You might also need to enable debug logging on the NFS level. Let me know if you can find anything there.
Timeout reading mirror data and similar errors.
These are UI-specific errors caused by a network failure. Connection is getting lost somewhere between your browser and a Bitbucket node. Could you elaborate more on your current setup: what load balancer do you use for the Bitbucket cluster? Can you try accessing a specific Bitbucket note bypassing the balancer?
You also mentioned that you’re using a VPN for accessing Bitbucket DC, is it possible to access a mirror settings page bypassing the VPN? Could you try accessing it from some other machine?
Is the timeout error steadily reproducible for all the browsers on your machine?
