HDFS-17937. Limit fsimage transfer concurrency during checkpoint with multiple NameNodes.

[huangzhaobo99]

Description of PR

JIRA: https://issues.apache.org/jira/browse/HDFS-17937

This patch adds dfs.namenode.checkpoint.parallel.upload.max-threads to cap fsimage upload concurrency in StandbyCheckpointer when parallel upload is enabled.
The limit is needed to avoid excessive simultaneous fsimage uploads in multi-NameNode (9 Observer NN) HA deployments, which can otherwise increase CPU/Network/Disk pressure and impact checkpoint/upload stability.
Effective concurrency is now min(remoteNNAddresses.size(), max-threads), with config/docs updates in DFSConfigKeys, CheckpointConf, and hdfs-default.xml, plus validation that max-threads >= 0.

How was this patch tested?

Added TestStandbyCheckpoints#testCheckpointParallelUploadRespectsMaxThreads UT to verify observed upload concurrency does not exceed configured max threads.

[huangzhaobo99]

@Hexiaoqiao @ayushtkn @tomscut , could you please help review this patch?
This change introduces a configurable cap on checkpoint fsimage upload concurrency.
It targets deployments (for example, 2 NameNodes + 9 Observers), where unconstrained parallel upload can create too many concurrent transfers.
With this cap, we can keep parallelism while preventing excessive upload pressure.

[Hexiaoqiao]

Thanks @huangzhaobo99 for your works. Make sense to me. One concern, If the fsimage files may amass to large numbers because it can't upload to other namenodes on time when enable this configuration to end user by default?

[huangzhaobo99]

Thanks @huangzhaobo99 for your works. Make sense to me. One concern, If the fsimage files may amass to large numbers because it can't upload to other namenodes on time when enable this configuration to end user by default?

@Hexiaoqiao Thanks for review.

I updated the concurrency policy in code as follows:

1. When parallel upload is enabled and maxThreads == 0 (default), poolSize is set to remoteNNAddresses.size(), preserving the original behavior.
2. When maxThreads > 0, poolSize is set to Math.min(remoteNNAddresses.size(), maxThreads), so concurrency is capped by the configured maximum.

Therefore, when there are many Observer NN nodes, concurrency can be limited based on the actual deployment scenario, It will not easily lead to fsimage amass.

[Hexiaoqiao]

My bad, I want to say that the checkpoint could not execute as expect because checkpoint time cost will enlarge.
Consider case: 1 ANN + 1 SBN + 6 Observer, with configuration checkpoint every hour and maxThreads=1 when checkpoint and fsimage file size is 100GB. The whole time cost could be like the following,
100GB*(1024MB/GB)/(100MB/s) * 8 =136min where 8 = 1 (dump) + 7(upload)
Here 136min>60min, and checkpoint will postpone about 2 hours later which is not expect, right?
Thanks.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	27m 43s		trunk passed
+1 💚	compile	0m 53s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	compile	0m 57s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	checkstyle	0m 59s		trunk passed
+1 💚	mvnsite	1m 11s		trunk passed
+1 💚	javadoc	0m 54s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 52s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	2m 40s		trunk passed
+1 💚	shadedclient	19m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 47s		the patch passed
+1 💚	compile	0m 42s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 42s		the patch passed
+1 💚	compile	0m 41s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 41s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 43s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 219 unchanged - 0 fixed = 220 total (was 219)
+1 💚	mvnsite	0m 50s		the patch passed
+1 💚	javadoc	0m 43s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 40s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	2m 32s		the patch passed
+1 💚	shadedclient	22m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	180m 10s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		265m 39s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList

Subsystem	Report/Notes
Docker	ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8559/4/artifact/out/Dockerfile
GITHUB PR	#8559
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 223ec2e157a4 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 45e197d
Default Java	Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8559/4/testReport/
Max. process+thread count	3946 (vs. ulimit of 10000)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8559/4/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

[huangzhaobo99]

@Hexiaoqiao Yes, this can happen. This is expected, since the concurrency is configured to a low value, which increases checkpoint execution time.
In this scenario, we should set the parallelism to 3 rather than simply matching the number of remote NNs.
(Using 7 parallel threads is too high and unnecessary, This is also the main problem that this patch aims to solve)

[Hexiaoqiao]

@Hexiaoqiao Yes, this can happen. This is expected, since the concurrency is configured to a low value, which increases checkpoint execution time. In this scenario, we should set the parallelism to 3 rather than simply matching the number of remote NNs. (Using 7 parallel threads is too high and unnecessary, This is also the main problem that this patch aims to solve)

Got it. However we have changed the configuration dfs.namenode.checkpoint.period logic here, I think we need tell the truth to end users when release at least. Thanks.