HADOOP-19450: [ABFS] Rename/Create path idempotency client-level resolution

[bhattmanish98]

Description

CreatePath and RenamePath APIs are idempotent as subsequent retries on same resource don’t change the server state[1]. However, when client experiences connection break on the CreatePath and the RenamePath APIs, client cannot make sense if the request is accepted by the server or not.

On connection failure, the client retries the request. The server might return 404 (sourceNotFound) in case of RenamePath API and 409 (pathAlreadyExists) in case of CreatePath (overwrite=false) API. Now the client doesn’t have a path forward. Reason being, in case of CreatePath, client doesn’t know if the path was created on the original request or the path was already there for some other request, in case of RenamePath, client doesn’t know if the source was removed because of the original-try or it was not there on the first place.

Proposed Solution

Driver will send addition header "x-ms-client-transaction-id" which will store by the server. In case first call fails because of time out and retry happens and server throw source not found (in case of rename) and path already exist (in case of create call). Driver will do list call on the path and check whether the "x-ms-client-transaction-id" returned by server same as what driver has at its end. In such case driver will return success to the caller.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 7 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 13s		trunk passed
+1 💚	compile	0m 41s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 35s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	checkstyle	0m 32s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 35s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	1m 12s		trunk passed
+1 💚	shadedclient	36m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 34s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 34s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 20s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 💚	mvnsite	0m 32s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	1m 16s		the patch passed
+1 💚	shadedclient	37m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 42s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		131m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7364/1/artifact/out/Dockerfile
GITHUB PR	#7364
JIRA Issue	HADOOP-19450
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 7d4c7cf69c8d 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / e2eba21
Default Java	Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7364/1/testReport/
Max. process+thread count	541 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7364/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 7 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 36s		trunk passed
+1 💚	compile	0m 49s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 37s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	checkstyle	0m 33s		trunk passed
+1 💚	mvnsite	0m 40s		trunk passed
+1 💚	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 31s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	1m 10s		trunk passed
+1 💚	shadedclient	37m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 33s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 20s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 💚	mvnsite	0m 31s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 25s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	1m 6s		the patch passed
+1 💚	shadedclient	38m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 42s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		131m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7364/2/artifact/out/Dockerfile
GITHUB PR	#7364
JIRA Issue	HADOOP-19450
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 272fc60f98e1 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 086f244
Default Java	Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7364/2/testReport/
Max. process+thread count	635 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7364/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.