Import fails, "error: svn: E215000: No matching credentials found."

SubGit
#1

After some successful experiments with SubGit on a personal system about a year ago, I’m trying to use Subgit for the first time on a system where I work. I’m having some difficulties I don’t recall encountering then.

I’m setting up a mirror with SubGit 3.3.11, on a server running GitLab 13.9. The subversion server and client are 1.10.2, the OS is RHEL 8. I’ve created an empty GitLab project using GitLab’s root user.

The configure step appears to work correctly. I then edit the subgit/config file in the repository directory, specifying the URL like “svn+ssh://svnserver/path/repository” and then specify userName, sshKeyFile, and sshKeyFilePassphrase in the [auth “default”] section. I’ve commented all other lines in this section.

Then when attempting the install step, subgit responds with “INSTALLATION FAILED” followed by “error: svn: E215000: No matching credentials found.”

It seems to resemble this old issue, but the solution described there (removing the passphrase from the key) doesn’t work.

I’ve tried putting the username in the URL rather than the userName field, no difference in the result.

I’ve tried using the private key to ssh to the Subversion server manually (as the same user invoking subgit) and this works, after supplying the passphrase. This confirms that the user has access to the key file, it has the correct file permissions, the user on the Subversion server has its authorized_keys set up correctly, etc.

I’ve tried creating a ~/.ssh/config file with a host entry for the Subversion server, with an IdentityFile pointing to the private key. This works, and allows me to use svn on the command line (also requiring that I provide the passphrase). I’ve tried using subgit with and without this file in place. No difference in the result.

Looking at the syslog on the subversion server, it appears that no actual connection or login attempt is being made; it’s failing before getting to that point.

I’m not getting any logs from SubGit for this failed attempt. (All I see is the log from the configuration step). Is there some other way to get more information about what subgit is trying to do?

#2

Hi Chris,

may I ask you to check if the SVN repository is set to use local users database (like passwd file) and if yes, is there a user which name matches the name of the user you’re setting in SubGit configuration?

#3

The subversion repository is not configured with its own user database. We use svn+ssh:// exclusively, and all repositories are configured with a common group ownership (applied after repository creation). The user in question is a member of that group.

I have tested this separately with the same user, checking out from a repository to a temporary directory on the gitlab server, making a change, and committing it – all using the SSH key and passphrase.

When I do this manually, I see the normal SSH login process dumping messages to the syslog on the Subversion server, when the svn client creates an SSH tunnel and executes ‘svnserve -t’ through it.

However when I try the ‘subgit install …’ command, I don’t see such messages, suggesting it isn’t getting that far – it’s failing on some local problem.

#4

Hi Chris,

thank you for the detailed explanation.

You are completely right, it is most probable that SubGit does not even try to access the server over ssh as the E215000: No matching credentials found message means that SubGit has not found suitable credentials for the connection attempt, so yet, it may not even try to connect. The reason for that is not completely clear, however. I tried to re-produce this issue locally, but haven’t managed to do that, it worked as expected all the times. I must also note here that we did not get any reports about SSH functionality problems recently; in fact, this functionality was implemented years ago and was working fine since then. So I tend to assume that the issue is most probably caused by some local configuration issues, it rather looks like SubGit is not able to access the key file for some reason. Could you please check if the user git (which is supposed to run SubGit in GitLab environment) can access and use the key? Also, if possible could you please try to access the SVN repository using jsvn client from SVNKit package:

https://www.svnkit.com/org.tmatesoft.svn_1.10.3.standalone.zip

Try to run ‘jsvn log’ agains the same SVN repository over svn+ssh. And could you please share the SubGit configuration file you are using for this case?

#5

Thanks for the response, Ildar.

I am also guessing this is a configuration problem, though it’s strange that I didn’t have trouble when I tried this a year ago – as far as I recall, it “just worked” on my test platform. Unfortunately I don’t have that system available to analyze anymore, but I did keep the configuration file, which doesn’t even have the other lines in the “auth” section commented, as I had to do in this case.

At any rate, I’ve tried to simplify as much as possible from this point. I’m now using the “git” user ID, and connecting to a local Subversion repository on the same machine (‘cetus’) to eliminate network and user ID issues as a potential problem, though I am still using an svn+ssh:// URL. The ‘git’ user ID has the unix group assignment of the repository, and can do checkouts via the same URL and private key which I’ve put in a directory /var/opt/gitlab/git-data/id. The resulting behavior is unchanged even in this simplified case. I can SSH (now, to the same machine), I can use svn on the command line to check out and commit using the private key, but SubGit fails as before. I think we have a good setup now, to further troubleshoot the problem, probably something in my SubGit configuration.

I tried using SVNKit, and I hope the results aren’t an unrelated mystery – it doesn’t work.

The ‘git’ user doesn’t really have a normal home directory on the system (it’s /var/opt/gitlab), so I checked out a Subversion repository into /tmp using the svn client. I then used the svn client to commit a small change, update, and then ‘svn log -l 1’. All this works.

I then tried using SVNKit as you provided, which I put in /opt/svnkit – I ran ‘/opt/svnkit/bin/jsvn log’ and I get repeated requests for connection details, followed by an error. On the first set of requests, I provide the key filename and the passphrase, and it appears to remember these on subsequent requests.

[git@cetus test_sams]$ /opt/svnkit/bin/jsvn log -l 3
Authentication realm: svn+ssh://cetus
Username: git
Password for 'cetus' (leave blank if you are going to use private key):
Private key for 'cetus' (OpenSSH format) (leave blank to use a SSH agent): /var/opt/gitlab/git-data/id/gitmirror
Private key passphrase [none]:
Port number for 'cetus' [22]:
Authentication realm: svn+ssh://cetus
Username [git]:
Password for 'cetus' (leave blank if you are going to use private key):
Private key for 'cetus' (OpenSSH format) [/var/opt/gitlab/git-data/id/gitmirror]:
Private key passphrase [***********]:
Port number for 'cetus' [22]:
Authentication realm: svn+ssh://cetus
Username [git]:
Password for 'cetus' (leave blank if you are going to use private key):
Private key for 'cetus' (OpenSSH format) [/var/opt/gitlab/git-data/id/gitmirror]:
Private key passphrase [***********]:
Port number for 'cetus' [22]:
svn: E200015: Authentication cancelled
[git@cetus test_sams]$

During this process, I get no messages in the syslog indicating an attempt to connect. When I run the same command with the normal client (‘svn log -l 3’) I’m asked for the private key password, and then I see the expected series of syslog messages from systemd and sshd, and I get the Subversion log messages I requested.

I’m attaching the subgit configuration file, /var/opt/gitlab/git-data/repositories/@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git/subgit/config

It’s modified as little as possible from what was produced by the configure step.

config (10.2 KB)

#6

Hi Chris,

thank for the details and test results. They didn’t clarify the situation to very end, however; may I ask you to repeat the jsvn test with debug login enabled? Here is how the FINEST logging level can be enabled:

https://confluence.atlassian.com/fishkb/how-do-i-enable-debug-logging-for-svnkit-644579448.html

Also, could you please try to connect using the IP address instead of the hostname?

Thank you in advance.

#7

Hi Ildar,

I’ve tried what you’ve requested and everything else I can think of to get that to work, and I’m not having any luck. At first I thought that the svnkit log file (~/svnkit.0.log) wasn’t being created because the ‘git’ user that is created by the Gitlab omnibus install doesn’t have write access to its home directory (/var/opt/gitlab) by default. So I changed its home directory to a new one that it owns (var/opt/gitlab/git), but I still don’t see the log file after trying to run jsvn.

Here’s what I’ve tried:

  • I tried using the logging.properties.disabled file linked from that page.
  • I am not using Fisheye or Crucible, so steps 2 and 3 do not apply. The jsvn script itself appears to be looking for ‘logging.properties’ in the conf directory (/opt/svnkit/conf in my case), so I put the downloaded file, without the “.disabled” extension, in that directory.
  • Even though I’m not using Fisheye or Crucible, I still tried actually following the main instruction in the link you sent – I added “-Djava.util.logging.config.file=/opt/svnkit/conf/logging.properties” to the jsvn command line. Of course, I got “svn: E205000: invalid option: -D” as a result.
  • I also tried removing the downloaded file, then using the logging.properties.disabled file that was part of the svnkit archive you provided, copying it to ‘logging.properties’ as indicated in the comment at the top of the file. It appears to be functionally the same, so I didn’t expect this to make a difference.
  • I tried using strace to see if jsvn is attempting to create the log file, and found no reference to it. Curiously, I also saw no attempt for the process to read the logging.properties file.
  • Since the jsvn script does not contain a shebang line, and the Gitlab installer configures the git user with /bin/sh as its shell, I thought maybe there’s some part of the script that’s misbehaving because it’s not expecting Bourne shell. I tried running it under bash explicitly; no difference in behavior, and still no log file.

I also tried doing a new checkout of the test repository with an IP address in the svn+ssh:// URL instead of a hostname. I added the IP address as a host entry to my ~/.ssh/config to associate the private key with the address so it would work the same way. Again, the stock svn command works, jsvn fails as before.

Is there a more general set of instructions for getting SVNkit to leave a log file? I’m not sure how to explicitly specify a non-default logging.properties file when calling jsvn from the command line, given that my attempts described above did not work.

#8

An additional clue, though I’m not sure yet what it means.

I went back to using a different user (a regular user ID, ‘chris’), just to test SVNkit. I found that with an SSH agent (my normal operating method), SVNkit works – I can check out, update, fetch logs, and so forth.

With this user ID, if I make a copy of the private key that the ‘git’ user was using (/var/opt/gitlab/git-data/id/gitmirror) into ~/.ssh (so ‘chris’ has read access to it) and try to check out a repository from svn+ssh://git@cetus/path/to/repository, I get the same behavior as before – it asks me repeatedly for the same connection details, and then ultimately gives me “E200015 Authentication cancelled”

So, the behavior might be specific to that key, or to using password-protected keys from files in general (rather than from an agent). Still no logs. I’ll continue to experiment with it.

Update: Created a new SSH key pair (RSA type) and added the public key to authorized_keys for user ‘chris’, then tested it to log in via SSH, it worked. Tested it to do a checkout with jsvn, and it fails as before.

So then I did this again, but this time with no key password. Again, SSH logins succeed, jsvn fails, with the same repeated questions and E200015.

So SVNkit seems to fail in this way with any use of an SSH key from a file (password protected or otherwise), but succeeds when the key is provided with an agent.

#9

Hi Chris,

that’s very interesting. May I ask you what’s the SSH agent you are using?

#10

That’s an interesting question. I tend to use an agent passthrough on the machine I’m usually working on – that is, I log in to work machines using ‘ssh -A’ or the equivalent configuration in ~/.ssh/config, and ultimately the agent on my client computer is usually KeePassXC, or KeePass with the KeeAgent plugin. My successful test with my own existing key was with this approach.

That’s a fairly specific setup, so to make it more general I just checked without that agent passthrough, running ssh-agent locally (from OpenSSH) and then ssh-add to load the password-protected key I just created. SVNKit works, with the key loaded in the agent.

With the very same key not available via an agent but loading from a file, SVNKit fails.

I think we’re getting somewhere interesting, whether it’s related to the original Subgit problem or not. :-)

#11

Hi Chris,

thanks for the explanation.

Interesting is that I’m still not able to reproduce this issue locally :( both SubGit and SVNKit work well on our test server with the path to the private key provided, no matter if the ssh-agent is running or not. Apparently, some of the infrastructure specifics differ on your server and our machine, since from the SubGit configuration standpoint everything looks the same. I thought that the ssh-agent using might be bringing that difference, but I didn’t manage to get the same E215000 result, both SubGit and SVNKit worked as expected, they were able to communicate with SVN when the path is provided in the configuration file and they even worked well with the key provided through ssh-agent. The only case when I got the same result was the case when the git user (which is supposed to run SubGit and which I used for all the tests, including ssh-agent start) had no rights to read the private key – in this case no private key was in fact available, so the No matching credentials found appeared, which was in fact expected.
So I suggest trying to check the infrastructure and setup step by step to find that difference. First I suggest to get rid of any of the ssh agents for a while for testing purposes and stick only with the file path. Could you please switch off all the agents on the GitLab server, not only those started by git user, but started by any users; then make sure that git user is able to read that file (cat <the key file> maybe) and try to run jsvn log and SubGit install?

#12

Hi Ildar,

Apologies for the delayed response.

I have started from scratch, focusing on reproducing the problem with SVNkit on a new test VM, with a fresh install of CentOS 8. I have not done any further experiments on the Gitlab server, or with Subgit.

In my initial test, I took the following steps:

  • Create VM (libvirt/KVM), install CentOS 8 (basic workstation install), install subversion, java-1.8.0-openjdk, then install your SVNkit zip archive and symlink to /opt/svnkit. The new VM machine name is centaur. Local user name is ‘cr’. I connect via SSH -A and have a key visible, for example, with ‘ssh-add -l’.
  • Use ‘svn’ to check out a simple project called ‘launcher’ from our repository server, with my existing username there: ‘svn co svn+ssh://chris@cetus/svn/launcher/trunk launcher’ This uses the agent and key, and succeeds.
  • Use ‘/opt/svnkit/bin/jsvn log’ in the working copy directory ‘launcher’, again using the agent. It succeeds.
  • Disconnect from centaur, and then reconnect using ‘ssh -a cr@centaur’, disabling the agent forwarding. As expected, ‘ssh-add -l’ can’t connect to an authentication agent.
  • Use ssh-keygen to create a key, ssh-copy-id to copy it to chris@cetus, use ssh -i to force the use of that key, testing it to log into cetus. It succeeds.
  • ‘/opt/svnkit/bin/jsvn log’ goes through the same three repeating prompt cycles as before, ending in “svn: E200015: Authentication cancelled”.

Next, I decided to redo the test with a new Subversion server VM as well. Cetus is running on RHEL 8, but I created the new VM with CentOS. The new VM is ‘minotaur’.

  • Create minotaur VM, same configuration as centaur, basic CentOS 8 workstation install. Install subversion, create ‘software’ group and add my local user ‘cr’ to it.
  • Create /svn directory, create dump file of ‘launcher’ repository on cetus, create empty launcher repository on minotaur. Load dump file.
  • Recursively apply ‘software’ group and ‘g+rw’ permissions to repository directory.
  • Connect to centaur, again with agent forwarding disabled. ssh-copy-id private key created above to cr@minotaur and test with ssh to make sure it works.
  • On centaur, create Host entry for minotaur in ~/.ssh/config, to associate private key file (“IdentityFile”) with ‘minotaur’ host and test with Subversion client to make sure I can check out ‘svn+ssh://cr@minotaur/svn/launcher/trunk’. It works. (As expected, I have to specify the private key password with each svn operation.)
  • Test with /opt/svnkit/bin/jsvn, same prompt sequence and E200015 error as before.
  • Disconnect, reconnect with agent forwarding enabled. SVNkit works.

At this point, I think I have demonstrated this behavior of SVNkit conclusively, with no unusual preconditions or anything specific to our network or base software install. Just a fresh installation of CentOS 8 from their public mirrors, Subversion 1.10.2 on server and client VMs, and SVNkit from your zip file. I’ve used a local user (cr) on both VMs, to eliminate possible issues with the LDAP-based user (‘chris’) that I was using before. I’ve used a brand new SSH key, generated with ‘ssh-keygen -t rsa’. I think all of this should be reproducible on your end with an equivalent configuration.

This may be a tangent from the Subgit problem, but it seems like it may be related and it’s probably worth sorting out anyway.

For what it’s worth, here’s that prompt sequence again, from my most recent attempt:

[cr@centaur launcher]$ /opt/svnkit/bin/jsvn log
Authentication realm: svn+ssh://minotaur
Username: cr
Password for 'minotaur' (leave blank if you are going to use private key): 
Private key for 'minotaur' (OpenSSH format) (leave blank to use a SSH agent): /home/cr/.ssh/cr_cen1
Private key passphrase [none]: 
Port number for 'minotaur' [22]: 
Authentication realm: svn+ssh://minotaur
Username [cr]: 
Password for 'minotaur' (leave blank if you are going to use private key): 
Private key for 'minotaur' (OpenSSH format) [/home/cr/.ssh/cr_cen1]: 
Private key passphrase [*************]: 
Port number for 'minotaur' [22]: 
Authentication realm: svn+ssh://minotaur
Username [cr]: 
Password for 'minotaur' (leave blank if you are going to use private key): 
Private key for 'minotaur' (OpenSSH format) [/home/cr/.ssh/cr_cen1]: 
Private key passphrase [*************]: 
Port number for 'minotaur' [22]: 
svn: E200015: Authentication cancelled
[cr@centaur launcher]$
#13

Helli Chris,

have you provided SVNKit with the full path to the newly created private key when it asked the following question:

Private key for '<SVN server>' (OpenSSH format) (leave blank to use a SSH agent):
#14

Yes indeed.

And you’ll notice in the latter two of the three prompts in the sequence that I pasted at the bottom, it remembers and offers as a default, the filename I typed in the first time, “/home/cr/.ssh/cr_cen1”.

#15

Hi Chris,

thank you for all the data and explanations, we are looking into it, I\ll let you know as we get something.

#16

Hi Chris,

could you please advise what is the format of the SSH private key you were trying to use, OpenSSH or RSA? If that is OpenSSH, could you please try to convert it to RSA (using -m key_format option) and try the same with RSA key?

#17

Related issue on our issue tracker: https://issues.tmatesoft.com/issue/SVNKIT-759

#18

Also having this problem, and dmitry’s OpenSSH->PEM seemed to be part of the issue. That just moved me from E215000 to E170001.

@dmitry.pavlenko @ildar.khusainov I tried converting ssh keys to PEM format, confirmed I’m able to log into the svn server, but nothing I do seems to get it to work with svn+ssh.

I’ve created a pretty simple Docker setup for testing this

Unfortunately, this gets me a totally different error:

image
image880×201 19.8 KB

error: svn: E210002: There was a problem while connecting to svnserver:22
error: There was a problem while connecting to svnserver:22
error: Key exchange was not finished, connection is closed.
error: Cannot negotiate, proposals do not match.
error: Unexpected error has occurred; please report along with the logs (’/home/gituser/subgit-install-20220329-081219.zip’)
error: to https://support.tmatesoft.com/, thank you!

To use this, have two terminal prompts open:

$ git clone https://github.com/kfsone/subgit.git
$ cd subgit
$ docker compose up

You’ll be able to see output from the ssh server in this terminal, now switch to the other:

$ cd subgit
$ docker compose exec -u gituser -it gitserver /bin/bash

this will drop you into a prompt inside the second server, the target for the subgit install. From here, the svn host is called “svnserver”. Verify the ssh keys work:

gituser@gitserver$ ssh -i ~/.ssh/id-blank svnuser@svnserver
...
svnuser@svnserver$ exit
gituser@gitserver$ ssh -i ~/.ssh/id-keyed svnuser@svnserver
# passphrase is: keyed
...
svnuser@svnserver$ exit

Finally, go ahead and try and do the subgit thing, there’s a drop-in config in the gituser homedirectory: subgit/subgit-test-config at main · kfsone/subgit (github.com)

gituser@gitserver$ subgit configure svn+ssh://svnserver/svn/Project1 Project1.git
gituser@gitserver$ cp subgit-test-config Project1.git/subgit/config
gituser@gitserver$ more $_
gituser@gitserver$ subgit install Project1.git
<< errors >>
#19

Hi Oliver,

as far as I found out, the keys in your project are of ECDSA type that is another thing the old library does not support, that is the main cause for the problem in your case. It should also be resolved by the fix introduced in SVNKit 1.10.5, we are now working on this release, plan to make it available in a few days; but it will be SVNKit release, for SubGit new release we will need some more time, I’ll be keeping you informed on progress.

1 Like
#20

@ildar.khusainov I’ve changed the ssh key to rsa and dsa, I still see the same problem.

I added my command history so you can literally now just do this:

git clone https://github.com/kfsone/subgit && cd subgit && docker compose build && docker compose up -d && docker compose exec -it -u gituser gitserver /bin/bash

and start hitting the up arrow until you reach the first command (#1 on the right side)

to make changes:

docker compose down && docker compose build && docker compose up

image
image705×887 63.8 KB

image
image879×619 55.9 KB

image
image1011×433 65.5 KB 

subgit-svnserver-1  | -- Exporting ssh keys to ssh-keys volume.
subgit-svnserver-1  | -- Starting container-only sshd in the background
subgit-svnserver-1  | -- Tailing sshd log
subgit-svnserver-1  | Server listening on 0.0.0.0 port 22.
subgit-svnserver-1  | Server listening on :: port 22.
subgit-svnserver-1  | Accepted publickey for svnuser from 192.168.96.3 port 33824 ssh2: RSA SHA256:knvHcx8Ox0GUGeFo1HnCg5q14nTnwHGuTn5yEclvdKU
subgit-svnserver-1  | Received disconnect from 192.168.96.3 port 33824:11: disconnected by user
subgit-svnserver-1  | Disconnected from user svnuser 192.168.96.3 port 33824
subgit-svnserver-1  | Accepted publickey for svnuser from 192.168.96.3 port 33826 ssh2: RSA SHA256:MC+JVdGKXIyW1fiJlpZnIamw+qJk39G3hbCQyYNSfgU
subgit-svnserver-1  | Received disconnect from 192.168.96.3 port 33826:11: disconnected by user
subgit-svnserver-1  | Disconnected from user svnuser 192.168.96.3 port 33826
subgit-svnserver-1  | Unable to negotiate with 192.168.96.3 port 33828: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
subgit-svnserver-1  | Unable to negotiate with 192.168.96.3 port 33830: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]

here’s the .zip:
subgit-install-20220329-175011.zip (6.6 KB)