fix(certificatee): persist validated storage certs

[ananthb]

Summary

• persist validated Vault certificate bundles to HAProxy storage on every certificatee sync
• keep runtime updates gated on live expiry or serial mismatch
• add coverage for storage repair when runtime is already current and runtime update after storage on serial mismatch

No Linear ticket - incident follow-up.

[github-actions]

Code Coverage Report

Total Coverage: total: (statements) 43.8%

Coverage by function

github.com/vinted/certificator/cmd/certificatee/health.go:27:		newCertificateeHealthChecker		100.0%
github.com/vinted/certificator/cmd/certificatee/health.go:38:		certificateeHealthMaxSyncAge		75.0%
github.com/vinted/certificator/cmd/certificatee/health.go:46:		Check					87.5%
github.com/vinted/certificator/cmd/certificatee/health.go:78:		MarkEndpointSyncSuccess			80.0%
github.com/vinted/certificator/cmd/certificatee/health.go:88:		lastSync				100.0%
github.com/vinted/certificator/cmd/certificatee/helpers.go:9:		createHAProxyClients			0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:35:		main					0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:87:		maybeUpdateCertificates			0.0%
github.com/vinted/certificator/cmd/certificatee/main.go:103:		processHAProxyEndpoint			37.7%
github.com/vinted/certificator/cmd/certificatee/main.go:196:		setDataPlaneAPIVersion			100.0%
github.com/vinted/certificator/cmd/certificatee/main.go:206:		shouldUpdateForLiveExpiry		100.0%
github.com/vinted/certificator/cmd/certificatee/main.go:224:		shouldUpdateForSerialMismatch		100.0%
github.com/vinted/certificator/cmd/certificatee/main.go:234:		validateVaultCertificateForUpdate	100.0%
github.com/vinted/certificator/cmd/certificatee/main.go:238:		validateVaultCertificateForUpdateAt	88.9%
github.com/vinted/certificator/cmd/certificatee/main.go:258:		parseVaultLeafCertificate		75.0%
github.com/vinted/certificator/cmd/certificatee/main.go:267:		readVaultCertificateBundle		71.4%
github.com/vinted/certificator/cmd/certificatee/main.go:281:		syncCertificate				76.9%
github.com/vinted/certificator/cmd/certificatee/main.go:329:		buildPEMBundle				70.0%
github.com/vinted/certificator/cmd/certificatee/main.go:355:		endsWith				66.7%
github.com/vinted/certificator/cmd/certificator/main.go:28:		main					0.0%
github.com/vinted/certificator/cmd/certificator/main.go:112:		deleteExpiredVaultCertificate		0.0%
github.com/vinted/certificator/cmd/certificator/main.go:116:		deleteExpiredVaultCertificateAt		100.0%
github.com/vinted/certificator/pkg/acme/acme.go:27:			GetEmail				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:32:			GetRegistration				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:37:			GetPrivateKey				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:42:			NewClient				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:61:			setupClient				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:80:			setupAccount				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:118:			newAccount				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:130:			getAccountKey				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:159:			registerAccount				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:177:			recoverAccount				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:211:			saveAccount				0.0%
github.com/vinted/certificator/pkg/acme/acme.go:221:			saveKey					0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:18:	ObtainCertificate			0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:50:	GetCertificate				0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:63:	ParsePEMCertificate			0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:80:	DeleteCertificate			0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:85:	IsExpired				0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:91:	NeedsReissuing				0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:119:	arraysEqual				0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:133:	arrayContains				0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:142:	VaultCertLocation			0.0%
github.com/vinted/certificator/pkg/certificate/certificate.go:146:	storeCertificateInVault			0.0%
github.com/vinted/certificator/pkg/certmetrics/metrics.go:79:		StartMetricsServer			0.0%
github.com/vinted/certificator/pkg/certmetrics/metrics.go:104:		newHandler				100.0%
github.com/vinted/certificator/pkg/certmetrics/metrics.go:128:		PushMetrics				0.0%
github.com/vinted/certificator/pkg/config/config.go:73:			LoadConfig				0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:57:		Error					0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:63:		IsHTTPStatus				100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:68:		unexpectedStatusError			100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:77:		NewClient				100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:112:		NewClients				92.3%
github.com/vinted/certificator/pkg/haproxy/client.go:137:		Endpoint				100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:142:		doRequest				88.9%
github.com/vinted/certificator/pkg/haproxy/client.go:160:		parseAPITime				66.7%
github.com/vinted/certificator/pkg/haproxy/client.go:173:		getConfigVersion			66.7%
github.com/vinted/certificator/pkg/haproxy/client.go:238:		ListCertificates			100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:252:		ListCertificateRefs			90.9%
github.com/vinted/certificator/pkg/haproxy/client.go:294:		GetCertificateDetail			68.4%
github.com/vinted/certificator/pkg/haproxy/client.go:337:		UpdateCertificate			80.0%
github.com/vinted/certificator/pkg/haproxy/client.go:371:		UpdateStorageCertificate		90.9%
github.com/vinted/certificator/pkg/haproxy/client.go:390:		CreateCertificate			78.9%
github.com/vinted/certificator/pkg/haproxy/client.go:423:		EnsureStorageCertificate		50.0%
github.com/vinted/certificator/pkg/haproxy/client.go:441:		DeleteCertificate			92.9%
github.com/vinted/certificator/pkg/haproxy/client.go:465:		ExtractDomainFromPath			100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:485:		NormalizeDomainForVault			0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:492:		StorageCertificateName			100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:498:		IsExpiring				100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:510:		NormalizeSerial				100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:543:		Error					100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:547:		Info					0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:551:		Debug					100.0%
github.com/vinted/certificator/pkg/haproxy/client.go:555:		Warn					0.0%
github.com/vinted/certificator/pkg/haproxy/client.go:560:		toLogrusFields				85.7%
github.com/vinted/certificator/pkg/vault/vault.go:18:			NewVaultClient				0.0%
github.com/vinted/certificator/pkg/vault/vault.go:27:			TokenLookupSelf				0.0%
github.com/vinted/certificator/pkg/vault/vault.go:39:			KVWrite					0.0%
github.com/vinted/certificator/pkg/vault/vault.go:53:			KVDelete				0.0%
github.com/vinted/certificator/pkg/vault/vault.go:66:			KVRead					0.0%
github.com/vinted/certificator/pkg/vault/vault.go:87:			vaultFullPath				0.0%
total:									(statements)				43.8%