Hi Team,
We are using the SVN Mirror plugin to perform migration from SVN to Bitbucket and as observed, the plugin is leading to timeout issues while raising pull requests. Logs are given below, please help us to rectify the issue as soon as possible.
./svnmirror.log:8494:2021-11-09 03:07:19,629 [hook-callback:thread-1] INFO sg.hooks - Timeout waiting for task to be scheduled on a master node org.tmatesoft.subgit.stash.mirror.SgException: Timeout waiting for task to be scheduled on a master node
./svnmirror.log:8728:2021-11-09 03:26:46,031 [hook-callback:thread-1] INFO sg.hooks - Timeout waiting for task to be scheduled on a master node org.tmatesoft.subgit.stash.mirror.SgException: Timeout waiting for task to be scheduled on a master node
Thanks,
Moodys Team.
Hello.
I’m afraid those only records are not enough for the diagnostics, we would rather need full log for the investigation. Those records per se may be expected in logs depending on conditions, so the records presence in logs does not mean there’s a problem. Could you please advise what is the issue you have with the add-on? Are there any delays with commits or pull requests creation or merging?
Hi Ildar,
Yes, it’s delaying the pull requests and timed out with the mentioned error. To rectify the issue we had to restart the production instance of bitbucket. Please let us know if we can provide additional info.
Thanks,
Moodys Team.
Hi Moodys team,
yes, please collect the add-on logs from an affected repository. It can be done on the Support tab in the SVN Mirror add-on UI in the affected repository:
Another way to collect logs is to take them right from the Bitbucket node filesystem. We would need Bitbucket logs, global SVN Mirror add-on log and the add-on repository-specific logs:
-
Bitbucket log:
$BITBUCKET_HOME/log/atlassian-bitbucket.log
-
Global SVN Mirror add-on log:
$BITBUCKET_HOME/log/svnmirror.log
-
Repository-specific log:
$BITBUCKETHOME/data/subgit/repositories/REPO_ID/subgit/logs
REPO_ID here stands for the number of the repository, it can be found as the ‘Location on disk’ in the repository details.
Besides, could you please describe your setup in a little more details? First, is that a Bitbucket Datacenter cluster or a single stand-alone server? If it’s a cluster, how many nodes it includes? Could it be possible to test directly connecting to every particular node (bypassing the balancer)? If yes, please perform such test and check if the issue present on every node or only on some of them. Also, if that’s a cluster, please collect logs from every node, not only from a single one.
And regarding the issue itself, does the delay occur on the pull request creation or on a merge? Does it occurs always or only on particular occurrences? For example, do a single-commit requests works well and can the issue be related to the size (number of commits) of the request?
Hi Ildar,
I’m unable to upload the file as it’s more than a 1Gb, please let us know the alternative solution for it.
Thanks,
Ajjayya GM
Hi Ajjayya,
I’m afraid we don’t have a possibility to store that big file. Could it be possible to share it through some cloud service, like Google Drive?
Also, please answer my questions, too, those answers are also essential for the diagnostics.
Hi Ildar,
Is that a Bitbucket Datacenter cluster or a single stand-alone server?
→ This is in the cluster we do have 2 servers in the cluster.
If it’s a cluster, how many nodes it includes?
→ 2 Servers
Could it be possible to test directly connecting to every particular node (bypassing the balancer)? If yes, please perform such test and check if the issue present on every node or only on some of them. Also, if that’s a cluster, please collect logs from every node, not only from a single one.
→ Currently we are not facing the issue, so we can’t reproduce the issue again.
And regarding the issue itself, does the delay occur on the pull request creation or on a merge?
→ This issue occurs at the time of merging the pull request.
Does it occur always or only on particular occurrences? For example, do single-commit requests works well and can the issue be related to the size (number of commits) of the request?
→ This issue is related to the number of commits.
We don’t have access from Moodys network to outside to upload the files to other cloud drives.
Thanks,
Ajjayya GM
Hello Ajjayya,
thank you for your answers.
If that is a cluster of several nodes and the issue is volatile, it’s most probably is caused by communications problems between the nodes. The matter is that in a cluster environment SVN Mirror add-on selects one of the nodes as a master and passes tasks to that node for the actual execution. The user interface, on the other hand, may run on a secondary node and that is where the issue occurs: when the UI is connected to the master node everything works within that single node and the issue does not appear (which is happening right now); but when the UI is connected to the secondary node, the add-on intercepts the merge request and tries to send it to the master node for actual handling and sending to SVN, but for some reason this communication fails, that’s why those Timeout waiting for task to be scheduled on a master node errors appears. So the root cause is that there are problems with the communication between nodes and it should be resolved in order to resolve the add-on issue.
Hi Ildar,
Please let us know if we can work around this issue. Is there a way to have a sticky-session to be configured only for the merge request?
Thanks,
Ajjayya GM
Hi Ajjayya,
I’m afraid there is no such features in the add-on, it’s rather a thing that is supposed to be configured in infrastructure.In fact, SVN Mirror add-on does not require merge requests or UI actions to run on a particular node only; besides it chooses the master node from time to time more or less randomly, so it does not make much sense trying to bind it to a particular server. What SVN Mirror add-on does require is that nodes are able to communicate with each other so that it can schedule a task on a master node and that communication issue is what should be diagnosed in this case, I think.
