feat: add Redis Cluster mode support (GCP Memorystore, AWS ElastiCache) by pedrojreis · Pull Request #17 · ClickHouse/code-interpreter

pedrojreis · 2026-07-07T17:53:51Z

Overview

Adds opt-in Redis Cluster support to every service component. Standalone
Redis remains the default — existing deployments require zero configuration
changes and behave exactly as before.

Validated in production against Google Cloud Memorystore in cluster mode with
TLS and CA-certificate verification.

Motivation

The service previously constructed Redis connections with inline
new IORedis({ ... }) calls in four separate modules, each hardcoded to
standalone mode. Connecting to a clustered Redis (GCP Memorystore cluster, AWS
ElastiCache cluster) was impossible: the client would only ever reach a single
shard and fail with MOVED/CROSSSLOT errors under load.

This PR centralizes connection creation behind a single factory and teaches
every component to speak the Redis Cluster protocol when asked.

What's new

🔌 Cluster mode (opt-in, auto-detected)

Enable it either explicitly or implicitly:

# Explicit
USE_REDIS_CLUSTER=true
REDIS_HOST=node1.example.com:6379

# Auto-detected — a comma in REDIS_HOST turns on cluster mode
REDIS_HOST=node1:6379,node2:6379,node3:6379

🔐 TLS with CA-certificate validation

REDIS_TLS=true
REDIS_CA=/etc/redis-tls/ca.crt   # PEM file → full cert verification

When REDIS_CA is set it takes precedence and enables validated TLS.
REDIS_TLS=true on its own keeps the previous rejectUnauthorized: false
behaviour for backward compatibility.

🧩 BullMQ cluster-safety

Queue, Worker and QueueEvents receive a {codeapi} hash-tag prefix in cluster
mode so all BullMQ keys map to a single hash slot (a hard requirement for BullMQ
on Redis Cluster). Standalone deployments keep their existing key layout — no
migration needed.

New environment variables

Variable	Default	Description
`USE_REDIS_CLUSTER`	`false`	Force cluster mode. Also auto-enabled when `REDIS_HOST` contains a comma.
`REDIS_CA`	(unset)	Path to a PEM CA-cert file. Enables TLS with full certificate validation; takes precedence over `REDIS_TLS`.

Existing variables are unchanged and fully backward-compatible:
REDIS_HOST, REDIS_PORT, REDIS_PASSWORD, REDIS_TLS,
REDIS_USE_ALTERNATIVE_DNS_LOOKUP, REDIS_KEEP_ALIVE_MS.

Implementation

`service/src/redis-connection.ts` (new — single source of truth)

Export	Responsibility
`createRedisConnection(overrides)`	Returns `Redis \| Cluster` based on env; each caller passes its own retry / readyCheck overrides
`isClusterMode()`	`USE_REDIS_CLUSTER=true` or comma in `REDIS_HOST`
`parseRedisNodes()`	Parses `REDIS_HOST` into `[{ host, port }]` startup nodes
`buildTlsOptions()`	`REDIS_CA` → `{ ca }` (validated); else `REDIS_TLS=true` → `{ rejectUnauthorized: false }`; else no TLS
`bullmqPrefix()`	`'{codeapi}'` in cluster mode, `undefined` otherwise

Refactored clients

All four inline new IORedis({ ... }) blocks now call createRedisConnection():

queue.ts — shared BullMQ connection + prefix: bullmqPrefix() on Queue / QueueEvents
workers.ts — prefix: bullmqPrefix() on both Worker instances
egress-ledger.ts — mutation-connection pool made cluster-safe (Cluster has no .duplicate(), so a fresh createRedisConnection() is used instead)
tool-call-server.ts, file-server.ts — session-state clients

`service/src/service/replay-state.ts`

scanKeys() is now cluster-aware. ioredis.Cluster has no top-level
scanStream, so in cluster mode the helper fans out across every master node
via cluster.nodes('master') and streams SCAN on each. Masters own disjoint
hash-slot ranges, so results never overlap. This fixes the runtime crash:

TypeError: <client>.scanStream is not a function

`service/src/config.ts`

Adds the USE_REDIS_CLUSTER flag to the parsed env.

Helm chart (`helm/codeapi/`)

New values.yaml surface:

redis:
  cluster:
    enabled: false
    nodes: ""                        # "host1:6379,host2:6379,host3:6379"
  tls:
    enabled: false
    caSecretName: ""                 # Secret holding the CA cert
    caKey: "ca"
    caMountPath: /etc/redis-tls/ca.crt
  useAlternativeDnsLookup: false     # required for GCP Memorystore cluster TLS

New _helpers.tpl templates — codeapi.redis.clusterEnabled,
codeapi.redis.tlsEnv, codeapi.redis.caVolume, codeapi.redis.caVolumeMount
— are wired into all five component Deployments, including mounting the CA cert
from a Secret into each pod.

`service/.env.example`

Documents every new variable with inline guidance.

Tests

New service/src/redis-connection.test.ts — 18 unit tests, no live Redis required:

Suite	Coverage
`parseRedisNodes`	single host, embedded port, comma list, whitespace trimming, default fallback
`isClusterMode`	explicit flag, comma auto-detect, standalone
`buildTlsOptions`	no TLS, `REDIS_TLS` only, `REDIS_CA` file read, CA precedence over `REDIS_TLS`, missing CA file
`bullmqPrefix`	standalone, cluster via flag, cluster via comma host

✓ 18 pass   redis-connection.test.ts
✓ 35 pass   egress-ledger / egress-gateway / replay-state (unchanged, still green)

Backward compatibility

✅ Standalone is the default — no env changes for existing deployments.
✅ REDIS_TLS=true without REDIS_CA keeps the prior rejectUnauthorized: false behaviour.
✅ BullMQ key prefixes are added only in cluster mode; standalone key layout is untouched.
✅ No breaking changes to any existing environment variable.

How to verify

cd service
bun test src/redis-connection.test.ts     # 18/18 pass

# render the Helm chart in cluster mode
helm template codeapi helm/codeapi \
  --set redis.enabled=false \
  --set redis.cluster.enabled=true \
  --set redis.cluster.nodes="n1:6379\,n2:6379\,n3:6379" \
  --set redis.tls.enabled=true \
  --set redis.tls.caSecretName=my-memorystore-secret

… handling * Refactor job processing in workers.ts for improved readability and maintainability. * Introduce Redis connection management in redis-connection.ts. * Add tests for Redis connection utilities in redis-connection.test.ts. * Implement TLS options handling for secure Redis connections. * Enhance error handling and logging throughout the job processing flow.

* Updated the project dependency to version 2.3.1. * Ensured compatibility with existing codebase. * Ran tests to verify functionality post-upgrade.

CLAassistant · 2026-07-07T17:54:03Z

All committers have signed the CLA.

danny-avila · 2026-07-22T01:40:32Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 418e509eb3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-22T01:51:07Z

+
+    if (isClusterMode()) {
+        const nodes = parseRedisNodes();
+        return new Cluster(nodes, {


Hash-tag replay keys before enabling Redis Cluster

When USE_REDIS_CLUSTER is true this returns an ioredis Cluster, but replay state still performs multi-key Lua/transactions over untagged keys such as exec_state:<id>, exec_result:<id>, and tool_history:<id> in setExecutionResultScript and commitToolHistoryAndState. In Redis Cluster those key names hash to different slots, so PTC completion/continuation paths fail with CROSSSLOT once cluster mode is enabled; hash-tag the per-execution keys (or avoid multi-key ops) before constructing Cluster clients here.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-22T01:51:07Z

 {{- if .Values.redis.enabled }}
 {{- printf "%s-redis-master" .Release.Name }}
+{{- else if and .Values.redis.cluster.enabled .Values.redis.cluster.nodes }}
+{{- .Values.redis.cluster.nodes }}


Probe one Redis startup node in cluster mode

With the documented Helm setting redis.cluster.nodes="n1:6379,n2:6379,n3:6379", this helper renders the entire comma-separated list into the worker wait-for-redis initContainer command (nc -z {{ include "codeapi.redis.host" . }} {{ include "codeapi.redis.port" . }}). nc -h documents the syntax as [destination] [port], so the init container tries to resolve a literal destination like n1:6379,n2:6379,n3:6379 and never lets worker pods start; render/probe a single startup node or skip this probe for cluster mode.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-22T01:51:07Z

+**Redis Cluster mode (GCP Memorystore cluster, AWS ElastiCache cluster).** Set
+`redis.cluster.enabled=true` and provide the comma-separated startup nodes in
+`redis.cluster.nodes`. The chart will set `USE_REDIS_CLUSTER=true` and populate
+`REDIS_HOST` with the node list on every component. For TLS with a CA certificate


Document disabling the bundled Redis for clusters

Following this new cluster-mode README snippet with the chart defaults leaves redis.enabled as true; codeapi.redis.host then gives precedence to the bundled standalone Redis service while USE_REDIS_CLUSTER=true, so the pods instantiate an ioredis Cluster client against <release>-redis-master and ignore the provided startup nodes. Include redis.enabled=false in the example or make the helper/validation reject this conflicting configuration.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-22T01:51:07Z

-  ...(useAltDnsLookup
-    ? { dnsLookup: (address: string, callback: (err: Error | null, addr: string) => void): void => callback(null, address) }
-    : {})
+const redis = createRedisConnection({


Hash-tag tool-call session keys before Cluster cleanup

When this connection becomes an ioredis Cluster, the blocking-mode delete path still gathers tool_call:* keys and calls redis.del(...keys) for a session. A normal session can have multiple untagged keys (session, call, result/error) that hash to different slots, so cleanup fails with CROSSSLOT or misses keys under Redis Cluster; tag the tool-call keys by execution id or delete them one key/node at a time.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-22T01:51:07Z

+{{- if .Values.redis.tls.enabled }}
+- name: REDIS_TLS
+  value: "true"


Gate Redis TLS env on external Redis

The new values block says Redis TLS is for external managed Redis and is ignored when the bundled Redis is enabled, but this helper emits REDIS_TLS=true whenever redis.tls.enabled is set. If an operator enables that TLS block while leaving the chart default redis.enabled=true, every component tries to speak TLS to the plain in-cluster Bitnami Redis service and fails to connect; either suppress these env vars when redis.enabled is true or configure TLS on the bundled subchart too.

Useful? React with 👍 / 👎.

pedrojreis added 2 commits July 7, 2026 17:55

chore(deps): upgrade dependency to version 2.3.1

05a2c7f

* Updated the project dependency to version 2.3.1. * Ensured compatibility with existing codebase. * Ran tests to verify functionality post-upgrade.

refactor(limits): improve Redis command handling and formatting

418e509

pedrojreis force-pushed the feature-memorystore-cluster branch 3 times, most recently from 0aabfff to 418e509 Compare July 7, 2026 21:36

chatgpt-codex-connector Bot reviewed Jul 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add Redis Cluster mode support (GCP Memorystore, AWS ElastiCache)#17

feat: add Redis Cluster mode support (GCP Memorystore, AWS ElastiCache)#17
pedrojreis wants to merge 3 commits into
ClickHouse:mainfrom
nosportugal:feature-memorystore-cluster

pedrojreis commented Jul 7, 2026

Uh oh!

CLAassistant commented Jul 7, 2026 •

edited

Loading

Uh oh!

danny-avila commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

pedrojreis commented Jul 7, 2026

Overview

Motivation

What's new

🔌 Cluster mode (opt-in, auto-detected)

🔐 TLS with CA-certificate validation

🧩 BullMQ cluster-safety

New environment variables

Implementation

service/src/redis-connection.ts (new — single source of truth)

Refactored clients

service/src/service/replay-state.ts

service/src/config.ts

Helm chart (helm/codeapi/)

service/.env.example

Tests

Backward compatibility

How to verify

Uh oh!

CLAassistant commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danny-avila commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

`service/src/redis-connection.ts` (new — single source of truth)

`service/src/service/replay-state.ts`

`service/src/config.ts`

Helm chart (`helm/codeapi/`)

`service/.env.example`

CLAassistant commented Jul 7, 2026 •

edited

Loading