Add detailed design for EXPLAIN COPY patch series (0001-0003)#4
Open
shinyaaa wants to merge 9 commits into
Open
Add detailed design for EXPLAIN COPY patch series (0001-0003)#4shinyaaa wants to merge 9 commits into
shinyaaa wants to merge 9 commits into
Conversation
Design document covering: - 0001: grammar (ExplainableStmt + CopyStmt), ExplainOneUtility dispatch, new explain_copy.c, DoCopy/BeginCopyTo refactoring, non-ANALYZE output - 0002: EXPLAIN ANALYZE COPY (query) TO executing the query while producing no COPY output - 0003: EXPLAIN ANALYZE COPY FROM with per-phase timing breakdown (input/insert/index/trigger), instrumentation plumbing, overhead mitigation, output schema, and test plan Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZujnAnhcNbVyWUn4dwJVu
Per review feedback, separate the behavior-preserving refactoring of copy.c/copyto.c (ProcessCopyTarget, CopyToTransformQuery) into patch 0001 so reviewers can verify 'no functional change' in isolation. Renumber the feature patches accordingly (grammar/dispatch -> 0002, ANALYZE COPY TO -> 0003, ANALYZE COPY FROM -> 0004) and document the split policy: pure code motion gets its own patch, while API additions with no in-tree caller (ExplainOnePlan parameter, instrumentation setter) stay with the feature that uses them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZujnAnhcNbVyWUn4dwJVu
Replace the loose 'wrap call sites with INSTR_TIME pairs' description with a concrete API: a CopyFromPhase enum, a phase_time[] array in CopyFromInstrumentation, and static inline CopyFromInstrStartPhase()/CopyFromInstrStopPhase() helpers in copyfrom_internal.h. Document the non-nesting invariant (single phase_start, Assert-checked), error-path behavior, the exact insertion points in copyfrom.c, and the rejected alternative of reusing instrument.c's InstrStartNode/InstrStopNode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZujnAnhcNbVyWUn4dwJVu
…BOSE) Add a design-rationale section comparing the EXPLAIN extension against the alternative of a COPY option emitting a NOTICE summary: role consistency with DML statements, structured output formats, orthogonal composition with existing EXPLAIN options, result-set vs NOTICE output channel, plan display for COPY (query) TO, and the no-execution EXPLAIN mode. Also records the counterarguments (STDIN coverage, VACUUM VERBOSE precedent, 'EXPLAIN is for plans') with rebuttals for the proposal mail. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZujnAnhcNbVyWUn4dwJVu
Add three groups of additional counterarguments with rebuttals: alternatives (pg_stat_progress_copy, file_fdw workaround, profilers, cumulative stats), semantics/UX (\copy-over-STDIN practicality, phase sum vs Execution Time gap, observer effect, ANALYZE side effects, partial support matrix), and implementation/maintenance (auto_explain gap, pg_stat_statements consistency, hot-loop instrumentation rot, future parallel COPY). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZujnAnhcNbVyWUn4dwJVu
Extract the setup portion of DoCopy() -- permission checks, opening and locking the target relation, WHERE clause transformation, column privilege checks, and the RLS conversion to a query-based COPY -- into ProcessCopyTarget(). Likewise extract the parse analysis, rewrite and validation of the source query of COPY (query) TO from BeginCopyTo() into CopyToTransformQuery(). No functional changes. These functions will be reused by an upcoming patch adding EXPLAIN support for COPY. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZujnAnhcNbVyWUn4dwJVu
Add CopyStmt to ExplainableStmt in the grammar and dispatch it from ExplainOneUtility() to the new ExplainCopyStmt() in explain_copy.c. Plain EXPLAIN never executes the COPY: it performs the same permission checks and preparatory transformations as execution would (reusing ProcessCopyTarget()), then prints a description of the COPY operation: target relation, format, source/target and non-default options. For COPY (query) TO, the execution plan of the source query is shown as well; to print the COPY details inside the query's output group, ExplainOnePlan() gains a CopyStmt parameter, following the precedent of its IntoClause parameter for CREATE TABLE AS. EXPLAIN ANALYZE of COPY is rejected for now; support for it is added separately for COPY (query) TO and COPY FROM by upcoming patches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZujnAnhcNbVyWUn4dwJVu
With ANALYZE, the source query of a query-based COPY TO is now planned and executed through the regular EXPLAIN machinery, reporting node timings, buffer usage and WAL usage as for any other query. The COPY output itself is not produced: no file is written, and no data is sent to the client, paralleling how EXPLAIN ANALYZE discards the rows a SELECT would return. This also means no COPY protocol messages are emitted, so COPY ... TO STDOUT needs no special treatment. EXPLAIN ANALYZE of COPY FROM and of table-based COPY TO remains unsupported for now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZujnAnhcNbVyWUn4dwJVu
With ANALYZE, COPY FROM is now executed (actually loading the data, like EXPLAIN ANALYZE INSERT) and EXPLAIN reports the row counts and a breakdown of where the time was spent: - Input Time: reading, parsing and converting the input data, measured around the CopyFromOneRow format callback so that custom COPY formats are covered as well - Insert Time: table_(multi_)insert or FDW inserts - Index Update Time: ExecInsertIndexTuples - per-trigger times, reusing the executor's trigger instrumentation by requesting es_instrument on the COPY's EState Buffer and WAL usage are reported from pgBufferUsage/pgWalUsage diffs. Rows excluded by the WHERE clause and rows skipped by ON_ERROR are also shown. The instrumentation is attached to the CopyFromState with the new CopyFromSetInstrumentation() function, keeping the BeginCopyFrom() API unchanged for external callers such as file_fdw. The per-phase timers are inline no-ops when no instrumentation is attached, so regular COPY pays only a branch per call site; with TIMING OFF only row counters are collected. EXPLAIN ANALYZE COPY FROM STDIN is rejected: the CopyInResponse protocol message would be sent while the client expects the result of the EXPLAIN statement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZujnAnhcNbVyWUn4dwJVu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design document covering:
new explain_copy.c, DoCopy/BeginCopyTo refactoring, non-ANALYZE output
producing no COPY output
(input/insert/index/trigger), instrumentation plumbing, overhead
mitigation, output schema, and test plan
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01EZujnAnhcNbVyWUn4dwJVu