SVN mirror Not in sync and in hung state

SVN Mirror
#1

Subgit mirror (3.4.3) on Bitbucket (5.8.3) has been working without problems but When ever we have patching happen on our servers Bitbucket starts, the mirror appears to start up but then hangs on every mirror and makes no sync events after the first that is scheduled . These issues is same as the issue we raised https://issues.tmatesoft.com/issue/SG-270 which keep on occuring when ever Bitbucket restarts and we have patching window for every 2 months which will make bit bucket restart.
Bit Bucket server info
Host is RHEL6 VM, 4 cores, 50gig ram.
$ uname -a
Linux stashap01 2.6.32-754.6.3.el6.x86_64 #1 SMP Tue Sep 18 10:29:08 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

Attached svn mirror logs, hopefully you guy’s provide us some feedback and steps to resolve future occurrences

#2

Hello Naveen

Thank you for reporting the issue and thank you for the logs.

According to the logs, there are many OOMs during connection to the database and it indeed may be the reason for the issue. The add-on version 3.4.3 that you are using may have memory leaks when it connects to the database; this problem has been repaired in the latest version 3.4.5, so I would recommend updating the add-on up to 3.4.5 as the first step. Also, we are implementing more changes affecting database connections, I’d recommend considering to update to the next version is it will be released (in a two or three weeks).

Also, as you run that many mirrors, I would recommend increasing the heap size if it wasn’t already done.
Additionally, it would be useful to show only active mirrors, as UI operations also require connections to the database; this behavior can be set by the ‘Administration - Global SVN Mirror Settings - Add-on Settings - Performance - Track Active Mirrors Only’ setting.

And, to collect more information in case if those steps don’t help, please enable debug logging prior to the next Bitbucket restart: ‘Administration - Global SVN Mirror Settings - Add-on Settings - Troubleshooting - Enable Debug Logging’. With this setting, the add-on writes much more information to the logs that allows us to catch the issue that may be unrecognizable in the regular logs.

#3

Hi @ildarhm , As you mentioned we upgraded to latest version 3.4.5 but the upgrade is taking long time to complete its keep on running from 40mins ,It looks like it is doing some cleanup stuff, usually how long it will take to upgrade, attached svnmiiror logs and screen shot of installing window and the heap size we have allocated is 10gig on bitbucket server

#4

Hi @ildarhm looks like that upgrade didn’t worked ,it keeps on running we waited for 7 hours to see if it get completed but as mentioned previous it is in hung state , we need to restart bitbucket to kill that upgrade.

So after restarting the mirror went into wired state , No able to even start , I have attached svn mirror logs after restarting Bitbucket and screen short of error and looks like it upgraded attached screen shot of that and not working…

#5

Hello Naveen.

Thank you for the logs.

It looks that the main problem is that the global add-on setting “Thread pool size” is set to “Unlimited”. That leads to a situation when all the configured mirrors can start synchronization simultaneously instead of being queued. Since you have 256 active mirrors, that leads to many connections to the database, many connections to SVN, etc., and that eventually leads to OOM that stops the add-on.

The solution is to decrease “Thread pool size”, it should limit add-on in resources and thus resolve the problem. There is a problem, however: this setting can only be changed in Bitbucket UI. Please decrease the setting value to 16 or 8 if you can access the aa-on global settings page (Administration - SVN Mirror - Global SVN Mirror Settings - Add-on Settings); otherwise, if the page cannot be open, I’m afraid that the only way to get it done is to temporarily disable all the mirrors.

The mirrors can be disabled through the add-on REST API; to disable the mirror on a repository with REPO_SLUG in a project with PROJECT_KEY, send the following POST request:

<Bitbucket URL>/rest/svn/1.0/projects/PROJECT_KEY/repos/REPO_SLUG?command=stop

I have added the file that consists all the mirrored repositories in a form ‘PROJECT_ID’_‘REPO_ID’; those are numeric IDs, so to disable the mirror you need first to clarify the RPOJECT_KEY and REPO_SLUG. IT could be done by the following GET requests:

<Bitbucket URL>/rest/api/1.0/projects

This one will show all the projects in the system, so you can take PROJECT_KEY by its ID. Then, having the PROJECT_KEY you can request for the projects in that project:

<Bitbucket URL>/rest/api/1.0/projects/PROJECT_KEY/repos

This request will return all the repositories in that particular project, so you can filter it by the REPO_ID to get REPO_SLUG.

So, the suggested steps are:

  • temporarily disable access to the Bitbucket or, at least, limit access to the mirrored repositories;
  • disable all the mirrors through REST API;
  • decrease “Thread pool size” in UI to 16 or even 8;
  • additionally, I’d recommend enabling “Track Active Mirrors Only” and “Enable Debug Logging”;
  • enable all the mirrors back.

Please let me know if you would need any help in developing a script for disabling/enabling the mirrors.
Also, if the proposed action plan with mirrors disabling is not acceptable, let me know, we’ll consider a possibility to create a special build for you.

#6

just in case, I have created a script that can be used to disable or enable mirrors. It awaits path to the projects/repositories listing file as an argument. To change script’s behavior from ‘disable’ to ‘enable’ change the ‘?command=stop’ to ‘?command=start’.

#7

So @ildarhm we need to disable all mirrors in bitbucket and then need to re enable to Fix the broken state of svn_mirror when upgraded to New version 3.4.5

#8

The main thing is the “Thread pool size” decreasing – it could even be done without disabling and re-enabling if the SVN Mirror UI is accessible in ‘Administration’ in Bitbucket. If the SVN Mirror add-on settings are not accessible in UI at this time – then yes, you would need to disable, change “Thread pool size” (and optionally the rest two settings) and then re-enable mirrors back.

#9

Hi @ildarhm The rest api call to stop/disable mirror is not working as expected, I’m getting internal error reponse as 500. The command I’m using POST http://BitBucket-url/rest/svn/1.0/projects/ESM/repos/microscott?command=stop

#10

Hello Naveen,

We’re now preparing a new build of the add-on that should fix the problem you’re experiencing. We’ll send you a link to the build as soon as possible, please expect it later today.

Kind regards,
Semyon Vadishev

#11

Thanks @semyon.vadishev , We looking out to get help. So now people not able to commit their code through bitbuket due to SVN mirror error issues . It would be helpful if we can Fix it today. or at least we can disable svn mirror.

#12

Attched svn mirror logs when Disabling SVN Mirror for BitBucket Server when we do it from UI and we also used rest API to disable still not working as mentioned above

#13

Hi @ildarhm @semyon.vadishev Update: Now we have disabled the svn mirror plugin from bitbucket. if possible can we have call tomorrow morning
at USA Central time 9:00 AM or any time after that or before, I’m not sure where you guys are located. if you can make it, we can discuss about the issues we are facing and steps to resolve the issues. you can call us at +1 402 431 7695

#14

@Naveen Hello, Naveen. We’ve prepared a new build of the add-on that we think should resolve the issues you experience:

  1. add the following system property to the JVM_SUPPORT_RECOMMEDNED_ARGS value in one of the Bitbucket Server script files (that depends on the Bitbucket Server version):

BITBUCKET_INSTALL_DIR/bin/setenv.sh
OR
BITBUCKET_INSTALL_DIR/bin/_start-webapp.sh

Corresponding line in the script file should be changed to:

JVM_SUPPORT_RECOMMENDED_ARGS=-Dsvnmirror.maxCoreRunnersCount=8

This option will enforce upper limit on the maximum number of threads used for synchronization. With the value of 8 we could be sure that number of threads running will not exceed JVM memory and threads limit.

  1. Download new build of the add-on from: https://subgit.com/files/translator-stash-3.4.5-2-package.jar
  2. Shutdown Bitbucket Server
  3. Start Bitbucket Server
  4. In Bitbucket Server UI, go to Administation | Manage Add-ons and click “Upload Add-on”
  5. Select translator-stash-3.4.5-2-package.jar to upload
  6. Enable SVN Mirror Add-on, new version should be started and it will consider property you’ve set in the script.

Regarding the call, it looks that 9am your time will be 4pm our time (we’re in Prague, GMT+1) we’re going to have a team meeting at that time and will call you to discuss the current state of the issue.

Thanks!

#15

Support email: support@subgit.com

#16

Thank you @semyon.vadishev For your help and be on call to troubleshooting things and providing information on svn mirror, Everything works fine now. we are working with teams to make svn mirror more stable to remove unsync errors. I will update ticket once done.

I appreciate your time.

#17

Thanks for the update, @Naveen. I’m glad to hear that.

On a related note: we decided to limit the thread pool size in the next version of the add-on, so the ‘Unlimited’ setting won’t result in a huge number of active threads connecting to the database.