Feat/watchdog tick metrics file

[stephenctw]

No description provided.

[endersonmaia]

Tested it using the instructions at https://github.com/cartesi/sequencer/blob/main/docs/watchdog/getting-started.md#path-b--interactive-sequencer--watchdog-two-terminals and when running tick I get unknown in the chain for the .prom file.

✗ cat /tmp/watchdog-state-devnet/status.prom
# HELP cartesi_watchdog_status Current watchdog compare state (1 = active).
# TYPE cartesi_watchdog_status gauge
cartesi_watchdog_status{app_address="0x7508F403A38a0cE53ca70F3DC4c150d614CE371C",chain="unknown",state="ok"} 1
cartesi_watchdog_status{app_address="0x7508F403A38a0cE53ca70F3DC4c150d614CE371C",chain="unknown",state="warning"} 0
cartesi_watchdog_status{app_address="0x7508F403A38a0cE53ca70F3DC4c150d614CE371C",chain="unknown",state="failed"} 0
# HELP cartesi_watchdog_last_tick_unix_seconds Unix time of the last completed tick.
# TYPE cartesi_watchdog_last_tick_unix_seconds gauge
cartesi_watchdog_last_tick_unix_seconds{app_address="0x7508F403A38a0cE53ca70F3DC4c150d614CE371C",chain="unknown"} 1783343927
# HELP cartesi_watchdog_exit_code Process exit code from the last tick.
# TYPE cartesi_watchdog_exit_code gauge
cartesi_watchdog_exit_code{app_address="0x7508F403A38a0cE53ca70F3DC4c150d614CE371C",chain="unknown"} 0

It should be 31337

✗ cast chain-id --rpc-url=http://127.0.0.1:37083
31337

[endersonmaia]

Another round of tests on my side, and now I'm getting this CURL error and I can't solve it.

The error message is not clear to me at where it's trying to connect to.

✗ ./watchdog/sequencer-watchdog tick
watchdog_step load watchdog checkpoint
watchdog_step fetch sequencer GET /finalized_state/inclusion_block
watchdog_step load watchdog checkpoint
watchdog_step fetch sequencer GET /finalized_state/inclusion_block
watchdog_step load watchdog checkpoint
watchdog_step fetch sequencer GET /finalized_state/inclusion_block
watchdog tick failed: [CURL-EASY][COULDNT_CONNECT] Could not connect to server (7)

At least I could see that the .prom file is populated with the state=warning = 1

✗ cat /tmp/watchdog-state-devnet/status.prom
# HELP cartesi_watchdog_status Current watchdog compare state (1 = active).
# TYPE cartesi_watchdog_status gauge
cartesi_watchdog_status{app_address="0x7508F403A38a0cE53ca70F3DC4c150d614CE371C",chain="unknown",state="ok"} 0
cartesi_watchdog_status{app_address="0x7508F403A38a0cE53ca70F3DC4c150d614CE371C",chain="unknown",state="warning"} 1
cartesi_watchdog_status{app_address="0x7508F403A38a0cE53ca70F3DC4c150d614CE371C",chain="unknown",state="failed"} 0
# HELP cartesi_watchdog_last_tick_unix_seconds Unix time of the last completed tick.
# TYPE cartesi_watchdog_last_tick_unix_seconds gauge
cartesi_watchdog_last_tick_unix_seconds{app_address="0x7508F403A38a0cE53ca70F3DC4c150d614CE371C",chain="unknown"} 1783357884
# HELP cartesi_watchdog_exit_code Process exit code from the last tick.
# TYPE cartesi_watchdog_exit_code gauge
cartesi_watchdog_exit_code{app_address="0x7508F403A38a0cE53ca70F3DC4c150d614CE371C",chain="unknown"} 1

[stephenctw]

@endersonmaia Hey Enderson — pushed fixes for both.

chain="unknown" — devnet exports didn't include CARTESI_WATCHDOG_BLOCKCHAIN_ID, and your existing state dir had no chain id from init. Now:

just devnet-for-watchdog exports CARTESI_WATCHDOG_BLOCKCHAIN_ID=31337
on tick we also resolve chain from env → persisted config → eth_chainId from the L1 RPC
You shouldn't need to re-init. Pull latest and run tick again — should show chain="31337". If you want to be explicit: export CARTESI_WATCHDOG_BLOCKCHAIN_ID=31337 before tick.

CURL error — improved the message to include the URL, e.g.: GET http://127.0.0.1:.../finalized_state/inclusion_block failed: [CURL-EASY][COULDNT_CONNECT] ...

Your second run looks like the sequencer wasn't reachable (stack stopped or wrong CARTESI_WATCHDOG_SEQUENCER_URL from stale exports). state=warning in .prom is correct for that — tick failed after retries but it's not a divergence.

Let me know if it works on your side after pulling.

[endersonmaia]

It's working. Just small comments.

I'm not sure if these two metrics are necessary, but won't block for it.

cartesi_watchdog_last_tick_unix_seconds

Be it pull based or push based, prometheus already awaits for a timestamp attached to every metric. Unless it's an information that's relevant to the sequencer itself.

cartesi_watchdog_exit_code

If this is already mapped to the status, it's redundant and not necessary.

Ex.:

exit_code = 0 -> state=ok
exit_code = 1 -> state=watning
exit_code = 2 -> state=failed

This would reduce the number of metrics we need to keep around and less code to manage.

[endersonmaia]

While testing, I noticed that the the services started by just devnet-for-watchdog stoped without showing eny logs, and that was the reason for me to get curl error from the tick.

Not sure if we should be worried, but it would be nive to have some logs about it and do some long duration tests just to be sure.

For ex., I let this running and left for lunch and came back trying to run a couple of ticks and got the error.

Since the port information was kept in the .json, I had to remove and run init again (could also have changed the port inside .json I think) after calling just devnet-for-watchdog again that change the ephemeral ports and probably (I assume) a new blockchain history/state.

[stephenctw]

@endersonmaia, thanks for the update. I'm going to investigate deeper into the issues you brought up!

Thanks for the review — good catch on the redundant gauges.

I dropped cartesi_watchdog_last_tick_unix_seconds and cartesi_watchdog_exit_code. Status is just state=ok|warning|failed (+ divergence_info on failed). Exit codes still map the same way for the process itself.

On the silent devnet-for-watchdog death: that was the local harness only waiting on Ctrl+C, so when the sequencer child died overnight/lunch the parent just sat there. Terminal 1 now polls the child, prints exit status + log path + a log tail, and exits. Also a 1-minute heartbeat so it's clearer the stack is still alive.

On ports after restart: yes — ephemeral ports + new Anvil history. Tick now honors CARTESI_WATCHDOG_SEQUENCER_URL from the environment even after init, so you can paste the new exports without editing config.json. Fresh Anvil history still needs a clean state dir + init (rm -rf /tmp/watchdog-state-devnet).

Pull latest and you should see the slimmed .prom + clearer stack exit logging.