Fix unnecessary (and sometimes wrong) casts between int and size_t #496

Job	Run time
address_ub_sanitizer	11m 18s
basic_clang	1m 0s
basic_gcc	56s
check_docs	1m 1s
check_update	1m 15s
check-ansi	44s
enable_non-default_options	48s
external-tests-misc	1m 38s
external-tests-providers	1m 18s
external-tests-pyca (1.51.0, 3.9)	1m 7s
full_featured	1m 0s
fuzz_tests	3m 0s
legacy	4m 54s
memory_sanitizer	18m 25s
minimal	41s
no-deprecated	1m 5s
no-legacy	50s
non-caching	13m 3s
no-shared-macos (macos-13)	1m 48s
no-shared-ubuntu	50s
out-of-readonly-source-and-install-macos (macos-13)	1m 36s
no-shared-macos (macos-14)	54s
out-of-readonly-source-and-install-ubuntu	48s
out-of-readonly-source-and-install-macos (macos-14)	56s
self-hosted	0s
threads_sanitizer	59s
	1h 11m 54s

Provide feedback