Skip to content

agent: post-render execute hook never runs on the first render after a restart #340

Description

@alekc

The agent skips the post-render execute command on the first render of every agent process, so a reload configured there does not run when the agent starts. That is harmless when the application boots alongside the agent, but it silently drops the reload in the case that matters most: a secret that rotated while the agent was down.

MonitorSecretChanges decides to render and then guards the hook separately:

if (existingEtag != currentEtag) || firstRun {

    tm.WriteTemplateToFile(processedTemplate, &secretTemplate, templateId)
    existingEtag = currentEtag

    if !firstRun && execCommand != "" {          // agent.go:1935
        log.Info().Msgf("executing command: %s", execCommand)
        err := ExecuteCommandWithTimeout(execCommand, execTimeout)
        ...
    }
    if firstRun {
        firstRun = false
        ...
    }
}

firstRun and existingEtag are both locals in MonitorSecretChanges (agent.go:1870 and :1871), which is spawned per template at agent.go:3460, and neither is persisted anywhere. So every agent process starts with firstRun true and an empty etag, renders unconditionally, and skips the hook. Restart the container, reboot the host, or edit the config, and you get a render with no reload.

Concretely: a secret rotates at 02:00 while the agent container happens to be restarting. The agent comes back at 02:01, writes the new value to disk, and never runs the reload command. The application keeps serving the old credential until the next rotation, whenever that is.

Easy to confirm from the log. On a cold start with an execute command configured, you get the template engine: secret template ... has been rendered and saved to path ... line and no executing command: line after it.

This compounds with #310, which reports that a shared etag on a multi-secret template suppresses re-renders, and observes that "only a full agent restart (firstRun=true, which forces an unconditional render) picks up the change". Taken together, the one render path that is guaranteed to happen is also the only one that never fires the hook.

There is a reasonable argument that an initial render is not a change and so needs no reload. If that is the intent, then the documentation is misleading: docs/integrations/platforms/infisical-agent.mdx presents execute.command as ./reload-app.sh with nothing to suggest it will not run at startup. Running it on first render seems the better fix, since a reload command is expected to be idempotent and there is currently no way to opt in, but documenting the exception would at least stop people relying on it.

Version 0.43.114.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions