hi Ildar,
No luck . (Masking url and passwords)
# curl -k -u 'abcd:abcd' -H "Content-Type: application/json" -H "X-Atlassian-Token:no-check" --data '{"enabled" : false, "interval" : 60}' "https://OUR_URL/rest/svn/1.0/config?name=longPollOptions"
#
After executing above CURL , this is what I see
FYI - Previous to executing this command
Hi Santosh,
could you please also add the -v cURL option, just to make sure the request went well:
curl -v -k -u…
What about the number of tabs, could it be the case you have too many of them opened? Also, could you please advise are Bitbucket Server pages available and work well and if only SVN Mirror add-on pages are affected?
Hi Ildar,
I have closed all other tabs and tried - no luck
Have asked my colleague from other location in India - same error for him .
Have asked my colleague from US - same error for him .
bitbucket works perfectly fine , only issue with this particular plugin page.
Here is the curl output with verbose option. (Masked sensitive infromation with **)
[root@]# curl -v -k -u 'USER:PASS' -H "Content-Type: application/json" -H "X-Atlassian-Token:no-check" --data '{"enabled" : false, "interval" : 60}' "https://url/rest/svn/1.0/config?name=longPollOptions"
* About to connect() to *** port 443 (#0)
* Trying **...
* Connected to *** port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=**,ST=CA,C=US
* start date: Jul 13 18:06:19 2020 GMT
* expire date: Jul 13 18:06:19 2023 GMT
* common name: d**
* issuer: **
* Server auth using Basic with user '**'
> POST /rest/svn/1.0/config?name=longPollOptions HTTP/1.1
> Authorization: Basic Q2hha2lTYTpDaGludHVANDM5
> User-Agent: curl/7.29.0
> Host: **
> Accept: */*
> Content-Type: application/json
> X-Atlassian-Token:no-check
> Content-Length: 36
>
* upload completely sent off: 36 out of 36 bytes
< HTTP/1.1 500
< Content-Type: application/json
< Vary: accept-encoding
< X-AREQUESTID: @66KSMYx586x240284x30
< X-AUSERID: 906
< X-AUSERNAME: **
< X-Content-Type-Options: nosniff
< Set-Cookie: ApplicationGatewayAffinity=e20670a2de0f64d827dc4697ec998195d0bf74ae97a76d4c4122e00ce9902ac7;Path=/;Domain=**
< Set-Cookie: ApplicationGatewayAffinityCORS=e20670a2de0f64d827dc4697ec998195d0bf74ae97a76d4c4122e00ce9902ac7;Path=/;Domain=**;SameSite=None;Secure
< Date: Mon, 12 Jul 2021 16:49:09 GMT
< Connection: close
< Content-Length: 0
<
* Closing connection 0
[root@]#hi Ildar,
Any update on this please ? This is what we still see.
We are kind of running out of schedule and this is important for us
Hi Santosh,
we are investigating this right now, I’ll be keeping you informed.
Meanwhile, we could use the REST API for the mirroring operations, too, could you please advise what the changes or operations do you need to perform in repositories?
Hi Santosh,
we continue investigating the issue, yet haven’t managed to re-produce it in our test environment. May I ask you to share Bitbucket and SVN Mirror logs with us to investigate? I’m afraid add-on’s Support ZIP feature is not much of a help currently, but it’s possible to collect it directly from the filesystem. Also, could you please advise what is your Bitbucket Server version and edition, how many node are in the cluster (if that is Datacenter cluster), and how did you install the add-on – have you installed it from Atlassian Marketplace or in some other way?
Hi Ildar,
We just want to import a project with gitgc as false in a specific bitbucket repo . This is what the REST API that we need for now.
Hi Santosh,
just duplicating the email I’ve sent you. Email and this forum are two ways we are using to provide support yet this forum is preferred one, so I suggest continuing here.
Here is the email:
we are still investigating the issue, yet as I wrote in the forum thread, we haven’t managed to re-produce the issue locally, so the reason is not yet clear enough.
It would be much of a help for us to have Bitbucket and SVN Mirror logs (both global and affected repository specific) along with the setup details –
Bitbucket version and edition, SVN Mirror add-on installation path, Bitbucket cluster configuration (how many node it has and do they reside in the same data center),
and also how do you access the Bitbucket – do you have a VPN or maybe an SSH tunnel and of there are any network limitation, like some TCP ports closed?
This information will help us much to understand the overall situation and find out the cause.
Meanwhile, since you have a possibility to communicate with Bitbucket using REST API, we can proceed configuring and running mirrors through add-on’s REST API.
Here is the API documentation:
TMate SubGit: SVN Mirror Add-On for Bitbucket Server REST API
To import a project with gitgc as false in a specific bitbucket repo, as you mentioned, follow the chapter 2 of the article.The gc setting can be set in the additional configuration options like this:
…
“config” : {
"svn.triggerGitGC" : false,
“svn.fetchInterval” : 60,
“translate.eols” : false|true,
“translate.ignores” : false|true
}…
Please let me know if you have need any help with the REST requests.
hi Ildar,
All of a sudden the plugin started working again and we have started the import. I’m still unsure why/how it happened.
Hi Santosh,
glad to know ti started to work.
It’s pretty strange, however, that it’s back without any actions or reconfigurations, it may worth to proceed with the investigation to find out the cause checking the data I mentioned earlier today. IT would also worth to check the network connection and environment as such an issue most probably is caused by network issues.
hi Ildar,
Sure. I have asked out bitbucket admin team to un-install/re-install the plugin and it started working. But we have lost 3 crucial days around this.
I will get back to you with my progress again and thanks for all the support.
Hi Ildar,
I get the following Input/Output error while importing.
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: Input/output error
svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SourceFile:112)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SourceFile:96)
at com.a.a.a.d.C.a(SourceFile:1893)
at com.a.a.a.d.C.a(SourceFile:1046)
at com.a.a.a.d.C.a(SourceFile:782)
at com.a.a.a.d.C.b(SourceFile:700)
… 24 common frames omitted
Caused by: java.io.IOException: Input/output error
at java.io.FileOutputStream.close0(Native Method)
I tried to “Stop” importing and it has been showing “cancelling” for almost 24 hours now.
Can you please let me know on how to proceed on this?
Thanks
Hi Ildar,
Any update on this please? We have hard deadlines and I request your suggestion.
Hello Santosh,
Ildar is on vacation these days, so I’ll do my best to fix the problem.
Input/Output error.
This is a generic error that comes from the file system. Unfortunately, it’s hard to say what exactly caused the failure, could your server admin scan through the NFS error log and see any relevant errors there?
Also, double-check that the disk space is sufficient on all the Bitbucket nodes you’re running. If you have debug logging enabled for both Bitbucket app and SVN Mirror add-on, I’d recommend disabling the debug logging, at least temporarily, since excessive logging may eat up some disk space.
Import cancellation is stuck.
The file system error got the import process stuck, so the cancellation is unable to complete. Unfortunately, the only way out is restarting the plugin: go to Administration | Manage apps | SVN Mirror for Bitbucket Server, click on Disable button there, wait until the plugin is unloaded and then enable it back.
Please let us know if restarting the plugin helps.
Hi Semyon,
Thank you for your reply.
I disabled the SVN Mirror for bitbucket server and enabled it back but I keep getting the following to check the status.
I checked the space and we have space on the nodes.
Can you please suggest the next actions.
Hello Santosh,
Thank you for the update. I’ve tried analyzing the log files you’ve provided us earlier and I believe you’re facing two major problems with our add-on:
Sporadic file system errors.
Looks like these errors occur during automatic mirror configuration and, as result, our add-on has to restart the process all over again. That leads to incomplete mirror configuration and the resulted branches mapping leads to inefficient import, e.g. fetching some tags as regular directories, which makes the import process extremely slow with enormous traffic between the SVN server and the Bitbucket node.
I think there’s a pattern in those file system errors: usually, our add-on creates a temporary file and then immediately deletes or renames it. This happens when a config file gets generated during automatic mirror configuration and also when a new object gets written into the objects database. I saw both types of errors in the log files.
There are two possible causes for such errors:
External process that interferes with short-lived files generated by our add-on. Please let me know if you’re using any kind of file scanners or any anti-virus software that theoretically may read those files in the background. It’s also possible that your operating system imposes some restrictions on the file system level that may interfere with our add-on, so let me know what OS you’re running. Finally, let me know if you’re running a Docker container for a Bitbucket node and if so, what image do you use for that, what version of Docker?
NFS-specific errors, e.g. network sync error on some short-lived files. Your server admin should be able to see those errors somewhere in the error log. You might also need to enable debug logging on the NFS level. Let me know if you can find anything there.
Timeout reading mirror data and similar errors.
These are UI-specific errors caused by a network failure. Connection is getting lost somewhere between your browser and a Bitbucket node. Could you elaborate more on your current setup: what load balancer do you use for the Bitbucket cluster? Can you try accessing a specific Bitbucket note bypassing the balancer?
You also mentioned that you’re using a VPN for accessing Bitbucket DC, is it possible to access a mirror settings page bypassing the VPN? Could you try accessing it from some other machine?
Is the timeout error steadily reproducible for all the browsers on your machine?
Hi Semyon,
I checked on other OS and the same error persists. Also there is no scanner that scans files bypassing VPN. Docker container is not used.
Tried checking multiple browsers with private/incognito modes as well and the issue still persists.
It is not possible to bypass VPN and access bitbucket. I tried from other machines as well.
Yes. The timeout error is steadily reproducible for all the browsers on my machine and other machines as well.
Can you please suggest what else can be done?
Hello Santhosh,
so far we are mostly out of ideas why this issue may happen - we had never encountered anything similar :(
What may give some clues is that “timeout” problem is not always reproducible - as far as I had understood it presented itself after you’ve restarted add-on. Also, a few months ago, that same issue resolved itself without any changes to the add-on or environment.
What I suppose might happen is that in your Data Center cluster of two nodes one SVN Mirror add-on doesn’t work properly on one of the nodes. If this would be the case, then balance loader may direct browser request to one node, and background request to another node (where add-on is not working or not replying properly).
To check this assumption I would suggest you to do the following, if that possible:
If problem persist, start the node you’ve shutdown back and disable another one. If my assumption is correct, this way we could find the node on which add-on is not working as expected and proceed from that.
In case issue persist even in single-node configuration then there might be some additional information in the browser dev console. To get that information:
With best regards,
Alexander Kitaev
Hi Alexander,
I have a quick question. Can we have an option to skip the current revision from migrating and proceed with the next one?
Please find the logs that show revisions “80381” through “80383” where the import stopped suddenly.
2021-08-01 17:52:35,835 sync - svn: E175002: Input/output error
svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’
2021-08-01 17:52:35,929 sync - svn: E175002: Input/output error
svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’ com.a.a.a.b.i: svn: E175002: Input/output error
svn: E175002: REPORT request failed on ‘/svn/!svn/vcc/default’
Is there a way that this revision can be skipped and then proceed to next revision and this specific revision can be copied manually?
I am in the process of bringing the node down and follow the steps as suggested by you.
Hello Santhosh,
Unfortunately, SVN Mirror can’t skip any specific revisions.
In this particular case, I recommend you skip the whole branch completely as it was copied from ^/branches/ directory. That means SubGit has to fetch the whole ^/branches/ directory in order to fetch ^/branches/GSPANN_ENHANCEMENTS_3. This is clearly the reason why your import gets stuck and then fails with I/O error.
You can skip the problematic branch with svn.excludeBranches option that you can specify along with the branches mapping:
[svn]
trunk = ...
...
excludeBranches = branches/GSPANN_ENHANCEMENTS_3
You can add this option and then proceed with the failed import.