Wrong behavior in SVNStatusEditor17 assembleStatus cause no svn commit

Hi,

if you are doing a checkout and change a file immediately (in a short amount of time) and the file has still the same size a svn commit does not see the change. The root cause is the if condition inside assembleStatus.

See https://svn.svnkit.com/repos/svnkit/tags/1.10.3/svnkit/src/main/java/org/tmatesoft/svn/core/internal/wc17/SVNStatusEditor17.java
and
https://svn.svnkit.com/repos/svnkit/tags/1.10.3/svnkit/src/main/java/org/tmatesoft/svn/core/internal/wc/SVNFileUtil.java

else if (ignoreTextMods ||
    (pathKind != null && 
            info.recordedSize != -1 &&
            info.recordedModTime != 0 &&
            SVNFileUtil.compareFileTimestamps(info.recordedModTime, fileTime) &&
            info.recordedSize == fileSize)) {
    text_modified_p = false;
}

The compareFileTimestamps inside got two different timestamps but inside SVNFileUtil#compareFileTimestamps a precision crop is done and returns “equal / true” but it isn’t equal.

Therefore the if condition is true and text_modified_p = false is set.

It can be reproduced easily if you are checkout something and do an edit very fast afterwards.

I’ll hope you’ll fix it, a System.sleep(1000) is a workaround after every checkout.

Greetings, Bernard

Hello Bernard,
thanks for reporting this. May I ask you to have a closer look at

SVNFileUtil.lastModifiedHasMicrosecondPrecision

constant? From your words I clearly conclude that your system supports microsend precision but static { ... } initializer near SVNFileUtil.java:2348 probably [incorrectly] discovers that this is not true. Is this so? There’re also other conditions on using full precision timestamps in the SVNFileUtil#compareFileTimestamps:


    public static boolean compareFileTimestamps(long microseconds1, long microseconds2) {
        if ((lastModifiedHasMicrosecondPrecision && java7BasciFileAttributesClazz != null) ||
                (SVNJNAUtil.isJNAPresent() && !isWindows)) {
            //only JNA on Unix and Java 9 have microseconds precision
            return microseconds1 == microseconds2;
        } else {
            //older Java and may have problems with precision (seconds only) as well as 'stat' command
            return (microseconds1 / 1000000) == (microseconds2 / 1000000);
        }
    }

I wonder which condition makes SVNKit [incorrectly] think that your system doesn’t support microsend precision timestamps? As you can see it depends on the 1) OS type; 2) JVM version.

If it’s the “constant” (not a constant actually) above that causing the problem, the work-around would be to set it to true.

In either case I would ask you to tell us which exactly condition fails and probably we will adjust the conditions to detect the precision correctly.

Hi, and thanks for the fast reply, here is an example call. I’ll hope everything is in sync because I’ve collected and synced these information during multiple runs:

boolean org.tmatesoft.svn.core.internal.wc.SVNFileUtil.compareFileTimestamps(long microseconds1, long microseconds2)
microseconds1 == 1643820964709620
microseconds2 == 1643820964741617

lastModifiedHasMicrosecondPrecision == false
java7BasciFileAttributesClazz == interface java.nio.file.attribute.BasicFileAttributes
isWindows == true
SVNJNAUtil.isJNAPresent() == true

which means the expression
(lastModifiedHasMicrosecondPrecision && java7BasciFileAttributesClazz != null) == false
which means the expression
(SVNJNAUtil.isJNAPresent() && !isWindows) == false

which means this will be executed: return (microseconds1 / 1000000) == (microseconds2 / 1000000)
and (microseconds1 / 1000000) == 1643820964
and (microseconds2 / 1000000) == 1643820964


it seems you are interested in the execution flow of:

final String javaVersion = System.getProperty("java.version");
if (javaVersion != null) {
    final String[] versionParts = javaVersion.split("\\.");
    if (versionParts.length >= 2) {
        try {
            final int majorVersion = Integer.parseInt(versionParts[0]);
            final int minorVersion = Integer.parseInt(versionParts[1]);

            if (majorVersion > MICROSECOND_PRECISION_JAVA_VERSION[0] ||
                    (majorVersion == MICROSECOND_PRECISION_JAVA_VERSION[0] &&
                            minorVersion >= MICROSECOND_PRECISION_JAVA_VERSION[1])) {
                lastModifiedHasMicrosecondPrecision = true;
            }
        } catch (NumberFormatException e) {
            //
        }
    }
}

MICROSECOND_PRECISION_JAVA_VERSION == [1, 9]
javaVersion == "1.8.0_222"
versionParts == [1, 8, 0_222]
majorVersion == 1
MICROSECOND_PRECISION_JAVA_VERSION[0] == 1

and therefore the if aborts because

majorVersion > MICROSECOND_PRECISION_JAVA_VERSION[0] == false
majorVersion == MICROSECOND_PRECISION_JAVA_VERSION[0] == true
minorVersion >= MICROSECOND_PRECISION_JAVA_VERSION[1] == false

Therefore
majorVersion > MICROSECOND_PRECISION_JAVA_VERSION[0] == false
(majorVersion == MICROSECOND_PRECISION_JAVA_VERSION[0] && minorVersion >= MICROSECOND_PRECISION_JAVA_VERSION[1]) == false
and the or is false and lastModifiedHasMicrosecondPrecision is not set to true.

Maybe because you’ve set private static final int[] MICROSECOND_PRECISION_JAVA_VERSION = {1, 9}; and compare with minorVersion >= MICROSECOND_PRECISION_JAVA_VERSION[1] but Java 1.8 supports it?


An additional information for long org.tmatesoft.svn.core.internal.wc.SVNFileUtil.getFileLastModifiedMicros(File file): file: C:\Users\bladenth\AppData\Local\Temp\junit4034958446396248862\junit8812005949469529045\DummyPojo.json
lastModifiedHasMicrosecondPrecision == false
java7BasciFileAttributesClazz == interface java.nio.file.attribute.BasicFileAttributes
SVNJNAUtil.isJNAPresent() == true

so we are going inside
//2. If this is not possible, try using JNA-based solution

which means we are going into
Long org.tmatesoft.svn.core.internal.util.jna.SVNJNAUtil.getLastModifiedMicros(File file)
isJNAPresent() == true
which executes
SVNLinuxUtil.getLastModifiedMicros(file)
but cLibrary == null and returns the method with null
timeMicros == null

java7BasciFileAttributesClazz == interface java.nio.file.attribute.BasicFileAttributes
so we are going inside

//3. If the previous approach didn't succeed, try Java 7 if it's available
which means we are executing
final Long timeMicros = getLastModifiedMicrosJava7(file);
path == C:\Users\bladenth\AppData\Local\Temp\junit4034958446396248862\junit8812005949469529045\DummyPojo.json
attrs == sun.nio.fs.WindowsFileAttributes@37095ded {creationTime == 132882945647096201, lastAccessTime == 132882945647416178, lastWriteTime == 132882945647416178}
time == FileTime {value == 1643820964741617, valueAsString == 2022-02-02T16:56:04.741617Z }

therefore this will be executed
final Object result = java7toTimeMethod.invoke(time, TimeUnit.MICROSECONDS);

and the result is
1643820964741617 which will be returned upwards up to assembleStatus

Thank you very much for your detailed description and sorry for not respoding for a while!

It looks like

  1. We have some prejudice about Java on Windows.
  2. Indeed we incorrectly think that Java 1.8 can’t give high resolution.

Here’s an interesting SO topic. According to it, things are complicated:

  1. For Oracle JDK, indeed, 1.8.162 doesn’t support high precision, while, for example, 9.0.4 does. On Debian it’s 1.8.0_292 which fixes the problem. Here’s the bug in the Oracle Bugs Database.
  2. For OpenJDK on solaris it depends not only on version but also on the compiler with which it was built (and in Oracle JDK they had the same problem and maybe it was the same code, the dependent compilation was fixed in Java 10). The fix in OpenJDK was done in Java 8 build 75 (tracker).
    Amazon Corretto 8 has this fix.

So it’s a bit complicated, in other words. I’ve just tried to read release notes for some releases. But I couldn’t find issue 8177809 among closed. On the other hand this list is not complete and “1.8u222” is missing in this list.

One of the solution would be to download different versions of JDK and find the fixing build by binary search but that would be too labourous. Probably it would be safe to assume that 1.8u292 fixes the problem for Oracle and 1.8 build 75 for OpenJDK.

May I ask you whether you have Java by Oracle or OpenJDK?

What about Windows, none of us it using it, so it’s always a challenge to find a Windows machine to test different scenarios on it. Regarding this issue, I believe it doesn’t depend on OS (except that conditional compilation on Solaris). So I’m thinking about removing “!isWindows” condition.

Hi, I’ve printed and obfuscated System.getProperties() for you, I’m using Zulu:

java.runtime.name: OpenJDK Runtime Environment
sun.boot.library.path: D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\bin
java.vm.version: 25.222-b10
java.vm.vendor: Azul Systems, Inc.
java.vendor.url: http://www.azulsystems.com/
path.separator: ;
jna.loaded: true
java.vm.name: OpenJDK 64-Bit Server VM
file.encoding.pkg: sun.io
user.country: DE
user.script: 
sun.java.launcher: SUN_STANDARD
sun.os.patch.level: 
java.vm.specification.name: Java Virtual Machine Specification
user.dir: D:\gitlab\SORRY_MUST_BE_OBFUSCATED
java.runtime.version: 1.8.0_222-b10
java.awt.graphicsenv: sun.awt.Win32GraphicsEnvironment
java.endorsed.dirs: D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\lib\endorsed
os.arch: amd64
java.io.tmpdir: C:\Users\bladenth\AppData\Local\Temp\
line.separator: 

java.vm.specification.vendor: Oracle Corporation
user.variant: 
os.name: Windows 10
sun.jnu.encoding: Cp1252
jnidispatch.path: C:\Users\bladenth\AppData\Local\Temp\jna--1064631222\jna4240027878632738988.dll
java.library.path: SORRY_MUST_BE_OBFUSCATED
javax.net.ssl.trustStore: SORRY_MUST_BE_OBFUSCATED
java.specification.name: Java Platform API Specification
java.class.version: 52.0
sun.management.compiler: HotSpot 64-Bit Tiered Compilers
os.version: 10.0
user.home: C:\Users\bladenth
user.timezone: Europe/Berlin
java.awt.printerjob: sun.awt.windows.WPrinterJob
file.encoding: Cp1252
java.specification.version: 1.8
java.class.path: D:\gitlab\SORRY_MUST_BE_OBFUSCATED\target\test-classes;D:\gitlab\SORRY_MUST_BE_OBFUSCATED\target\classes;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\reflections\reflections\0.9.10\reflections-0.9.10.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\google\guava\guava\15.0\guava-15.0.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\javassist\javassist\3.19.0-GA\javassist-3.19.0-GA.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\google\code\findbugs\annotations\2.0.1\annotations-2.0.1.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\opencsv\opencsv\3.9\opencsv-3.9.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\apache\commons\commons-lang3\3.5\commons-lang3-3.5.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\commons-beanutils\commons-beanutils\1.9.3\commons-beanutils-1.9.3.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\tmatesoft\svnkit\svnkit\1.10.1\svnkit-1.10.1.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\trilead\trilead-ssh2\1.0.0-build222\trilead-ssh2-1.0.0-build222.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\lz4\lz4-java\1.4.1\lz4-java-1.4.1.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\jcraft\jsch.agentproxy.connector-factory\0.0.9\jsch.agentproxy.connector-factory-0.0.9.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\jcraft\jsch.agentproxy.core\0.0.9\jsch.agentproxy.core-0.0.9.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\jcraft\jsch.agentproxy.usocket-jna\0.0.9\jsch.agentproxy.usocket-jna-0.0.9.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\jcraft\jsch.agentproxy.usocket-nc\0.0.9\jsch.agentproxy.usocket-nc-0.0.9.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\jcraft\jsch.agentproxy.sshagent\0.0.9\jsch.agentproxy.sshagent-0.0.9.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\jcraft\jsch.agentproxy.pageant\0.0.9\jsch.agentproxy.pageant-0.0.9.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\jcraft\jsch.agentproxy.svnkit-trilead-ssh2\0.0.9\jsch.agentproxy.svnkit-trilead-ssh2-0.0.9.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\de\regnis\q\sequence\sequence-library\1.0.3\sequence-library-1.0.3.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\net\java\dev\jna\jna\4.2.2\jna-4.2.2.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\net\java\dev\jna\jna-platform\4.2.2\jna-platform-4.2.2.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\tmatesoft\sqljet\sqljet\1.1.13\sqljet-1.1.13.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\antlr\antlr-runtime\3.4\antlr-runtime-3.4.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\com\github\lookfirst\sardine\5.6\sardine-5.6.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\slf4j\slf4j-jdk14\1.7.12\slf4j-jdk14-1.7.12.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\slf4j\slf4j-api\1.7.12\slf4j-api-1.7.12.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\apache\httpcomponents\httpclient\4.4.1\httpclient-4.4.1.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\apache\httpcomponents\httpcore\4.4.1\httpcore-4.4.1.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\commons-logging\commons-logging\1.2\commons-logging-1.2.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\commons-codec\commons-codec\1.9\commons-codec-1.9.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\commons-io\commons-io\2.4\commons-io-2.4.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\apache\ant\ant\1.9.4\ant-1.9.4.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\apache\ant\ant-launcher\1.9.4\ant-launcher-1.9.4.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\junit\jupiter\junit-jupiter\5.6.0\junit-jupiter-5.6.0.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\junit\jupiter\junit-jupiter-api\5.6.0\junit-jupiter-api-5.6.0.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\apiguardian\apiguardian-api\1.1.0\apiguardian-api-1.1.0.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\opentest4j\opentest4j\1.2.0\opentest4j-1.2.0.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\junit\platform\junit-platform-commons\1.6.0\junit-platform-commons-1.6.0.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\junit\jupiter\junit-jupiter-params\5.6.0\junit-jupiter-params-5.6.0.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\junit\jupiter\junit-jupiter-engine\5.6.0\junit-jupiter-engine-5.6.0.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\junit\platform\junit-platform-engine\1.6.0\junit-platform-engine-1.6.0.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\hamcrest\hamcrest-all\1.3\hamcrest-all-1.3.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\org\mockito\mockito-all\1.10.19\mockito-all-1.10.19.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\xmlunit\xmlunit\1.6\xmlunit-1.6.jar;D:\Maven\local_repository_SORRY_MUST_BE_OBFUSCATED3plus\net\lingala\zip4j\zip4j\1.3.2\zip4j-1.3.2.jar;C:\Users\bladenth\bladenthWorkspace\eclipse\configuration\org.eclipse.osgi\262\0\.cp;C:\Users\bladenth\bladenthWorkspace\eclipse\configuration\org.eclipse.osgi\260\0\.cp;C:\Users\bladenth\bladenthWorkspace\eclipse\configuration\org.eclipse.osgi\459\0\.cp;C:\Users\bladenth\bladenthWorkspace\eclipse\configuration\org.eclipse.osgi\263\0\.cp\lib\javaagent-shaded.jar
user.name: bladenth
java.vm.specification.version: 1.8
sun.java.command: org.eclipse.jdt.internal.junit.runner.RemoteTestRunner -version 3 -port 60231 -testLoaderClass org.eclipse.jdt.internal.junit5.runner.JUnit5TestLoader -loaderpluginname org.eclipse.jdt.junit5.runtime -test de.a.b.c.CommitTest:commandCommit_commitFailed
java.home: D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre
sun.arch.data.model: 64
user.language: de
java.specification.vendor: Oracle Corporation
awt.toolkit: sun.awt.windows.WToolkit
java.vm.info: mixed mode
java.version: 1.8.0_222
java.ext.dirs: D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\lib\ext;C:\WINDOWS\Sun\Java\lib\ext
sun.boot.class.path: D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\lib\resources.jar;D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\lib\rt.jar;D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\lib\sunrsasign.jar;D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\lib\jsse.jar;D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\lib\jce.jar;D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\lib\charsets.jar;D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\lib\jfr.jar;D:\SORRY_MUST_BE_OBFUSCATED\java\zulu8_64\jre\classes
java.vendor: Azul Systems, Inc.
file.separator: \
java.vendor.url.bug: http://www.azulsystems.com/support/
sun.io.unicode.encoding: UnicodeLittle
sun.cpu.endian: little
sun.desktop: windows
sun.cpu.isalist: amd64

Thanks a lot for the information!

I’ll update the conditions for the future SVNKit versions. For now setting

SVNFileUtil.lastModifiedHasMicrosecondPrecision

to ‘true’ is the best work-around.

Maybe a hint from my side: If you are already changing some logic, maybe implement a property like:
System.getProperty(“svnkit.fs.win32_retry_count”, “100”);

This allows the manual override of the autodetection. It help large scale projects where its hard to update to the newest version.

Thanks a lot and greetings from germany :)

Do you mean to have a property for?

SVNFileUtil.lastModifiedHasMicrosecondPrecision

E.g.

svnkit.fs.mtime_microsecond_precision

with “auto” (default), “true”, and “false”?

Exactly, maybe something like

svnkit.fs.mtime_microsecond_precision

allows as default (non set) with auto detection, or, if set, enabled/disabled. (TriState)

I’ve created an issue for that:

https://issues.tmatesoft.com/issue/SVNKIT-763

Also I’ve added the property and a setter at r10817 of SVNKit trunk. The fix will be included into the next release, though I can’t tell when it happens as we’ve released 1.10.4 quite recently.

If necessary, you can always build SVNKit from sources with

/gradlew svnkit:clean svnkit:build -x svnkit:javadoc -x svnkit:test svnkit-cli:clean svnkit-cli:build svnkit-distribution:clean svnkit-distribution:build

P.S.: thanks for greetng from Germany, Germany is to the north of me, to the south of me, and to the west of me.