Thank you for the logs.
It looks that the main problem is that the global add-on setting “Thread pool size” is set to “Unlimited”. That leads to a situation when all the configured mirrors can start synchronization simultaneously instead of being queued. Since you have 256 active mirrors, that leads to many connections to the database, many connections to SVN, etc., and that eventually leads to OOM that stops the add-on.
The solution is to decrease “Thread pool size”, it should limit add-on in resources and thus resolve the problem. There is a problem, however: this setting can only be changed in Bitbucket UI. Please decrease the setting value to 16 or 8 if you can access the aa-on global settings page (Administration - SVN Mirror - Global SVN Mirror Settings - Add-on Settings); otherwise, if the page cannot be open, I’m afraid that the only way to get it done is to temporarily disable all the mirrors.
The mirrors can be disabled through the add-on REST API; to disable the mirror on a repository with REPO_SLUG in a project with PROJECT_KEY, send the following POST request:
I have added the file that consists all the mirrored repositories in a form ‘PROJECT_ID’_‘REPO_ID’; those are numeric IDs, so to disable the mirror you need first to clarify the RPOJECT_KEY and REPO_SLUG. IT could be done by the following GET requests:
This one will show all the projects in the system, so you can take PROJECT_KEY by its ID. Then, having the PROJECT_KEY you can request for the projects in that project:
This request will return all the repositories in that particular project, so you can filter it by the REPO_ID to get REPO_SLUG.
So, the suggested steps are:
- temporarily disable access to the Bitbucket or, at least, limit access to the mirrored repositories;
- disable all the mirrors through REST API;
- decrease “Thread pool size” in UI to 16 or even 8;
- additionally, I’d recommend enabling “Track Active Mirrors Only” and “Enable Debug Logging”;
- enable all the mirrors back.
Please let me know if you would need any help in developing a script for disabling/enabling the mirrors.
Also, if the proposed action plan with mirrors disabling is not acceptable, let me know, we’ll consider a possibility to create a special build for you.