Skip to content

feat(cluster): original-vs-clustered comparison accessors (fixes #728) #734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FBumann merged 1 commit into from
Jul 21, 2026
Merged

feat(cluster): original-vs-clustered comparison accessors (fixes #728) #734

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 83 additions & 28 deletions docs/notebooks/08c-clustering.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
" Install with: `pip install \"flixopt[full]\"`\n",
"\n",
"!!! tip \"tsam_xarray\"\n",
" flixopt uses [tsam_xarray](https://github.com/FZJ-IEK3-VSA/tsam_xarray) for clustering,\n",
" flixopt uses [tsam_xarray](https://github.com/FBumann/tsam_xarray) for clustering,\n",
" which wraps [tsam](https://github.com/FZJ-IEK3-VSA/tsam). For advanced clustering options\n",
" (custom algorithms, weights, tuning), see the tsam_xarray documentation."
]
Expand Down Expand Up @@ -132,14 +132,39 @@
},
{
"cell_type": "markdown",
"id": "d5e6fd0c",
"source": "!!! note \"Which variables get clustered?\"\n By default, **all** time-varying inputs influence cluster assignments with\n equal weight (1.0). To see which variables `cluster()` will feed to tsam,\n call `list(flow_system.transform.cluster_inputs())` — it lists every\n variable with a `time` dim (constants included).\n\n To restrict clustering to a subset, pass explicit weights via\n `ClusterConfig(weights={...})`. Variables listed with weight `0` are still\n aggregated but don't influence cluster assignments; variables not listed\n keep the default weight of `1.0`. Example:\n\n ```python\n from tsam import ClusterConfig\n\n cols = list(flow_system.transform.cluster_inputs())\n target = 'HeatDemand(Q_th)|fixed_relative_profile'\n weights = {target: 1, **{v: 0 for v in cols if v != target}}\n\n fs_clustered = flow_system.transform.cluster(\n n_clusters=8, cluster_duration='1D',\n cluster=ClusterConfig(weights=weights),\n extremes=ExtremeConfig(method='new_cluster', max_value=[target]),\n )\n ```",
"metadata": {}
"id": "8",
"metadata": {},
"source": [
"!!! note \"Which variables get clustered?\"\n",
" By default, **all** time-varying inputs influence cluster assignments with\n",
" equal weight (1.0). To see which variables `cluster()` will feed to tsam,\n",
" call `list(flow_system.transform.cluster_inputs())` — it lists every\n",
" variable with a `time` dim (constants included).\n",
"\n",
" To restrict clustering to a subset, pass explicit weights via\n",
" `ClusterConfig(weights={...})`. Variables listed with weight `0` are still\n",
" aggregated but don't influence cluster assignments; variables not listed\n",
" keep the default weight of `1.0`. Example:\n",
"\n",
" ```python\n",
" from tsam import ClusterConfig\n",
"\n",
" cols = list(flow_system.transform.cluster_inputs())\n",
" target = 'HeatDemand(Q_th)|fixed_relative_profile'\n",
" weights = {target: 1, **{v: 0 for v in cols if v != target}}\n",
"\n",
" fs_clustered = flow_system.transform.cluster(\n",
" n_clusters=8, cluster_duration='1D',\n",
" cluster=ClusterConfig(weights=weights),\n",
" extremes=ExtremeConfig(method='new_cluster', max_value=[target]),\n",
" )\n",
" ```"
Comment thread
coderabbitai[bot] marked this conversation as resolved.
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8",
"id": "9",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -164,7 +189,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "9",
"id": "10",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -176,20 +201,20 @@
},
{
"cell_type": "markdown",
"id": "10",
"id": "11",
"metadata": {},
"source": [
"## Understanding the Clustering\n",
"\n",
"Access clustering metadata via `fs.clustering`. For full access to the underlying\n",
"[tsam_xarray ClusteringResult](https://github.com/FZJ-IEK3-VSA/tsam_xarray),\n",
"[tsam_xarray ClusteringResult](https://github.com/FBumann/tsam_xarray),\n",
"use `fs.clustering.clustering_result`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "11",
"id": "12",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -199,7 +224,37 @@
},
{
"cell_type": "markdown",
"id": "12",
"id": "13",
"metadata": {},
"source": [
"### Compare Original vs Clustered Profiles\n",
"\n",
"`clustering.compare()` returns a tidy `Dataset` (`original` vs `clustered`) for\n",
"**all** clustered variables, on the original time axis. flixopt bundles the\n",
"`.plotly` accessor, so stacking the two onto a `profile` dim and faceting is a\n",
"one-liner. `clustering.accuracy` reports the aggregation error."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "14",
"metadata": {},
"outputs": [],
"source": [
"print(fs_clustered.clustering.accuracy)\n",
"\n",
"(\n",
" fs_clustered.clustering.compare() # all variables; subset via .sel(variable=...)\n",
" .to_dataarray(dim='profile')\n",
" .plotly.line(x='time', color='profile', facet_row='variable')\n",
" .update_yaxes(matches=None) # variables have different scales\n",
")"
]
},
{
"cell_type": "markdown",
"id": "15",
"metadata": {},
"source": [
"### Apply Existing Clustering\n",
Expand All @@ -225,7 +280,7 @@
},
{
"cell_type": "markdown",
"id": "13",
"id": "16",
"metadata": {},
"source": [
"## Method 3: Two-Stage Workflow (Recommended)\n",
Expand All @@ -243,7 +298,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "14",
"id": "17",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -255,7 +310,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "15",
"id": "18",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -274,7 +329,7 @@
},
{
"cell_type": "markdown",
"id": "16",
"id": "19",
"metadata": {},
"source": [
"## Compare Results"
Expand All @@ -283,7 +338,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "17",
"id": "20",
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -332,7 +387,7 @@
},
{
"cell_type": "markdown",
"id": "18",
"id": "21",
"metadata": {},
"source": [
"## Expand Solution to Full Resolution\n",
Expand All @@ -344,7 +399,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "19",
"id": "22",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -355,7 +410,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "20",
"id": "23",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -377,7 +432,7 @@
},
{
"cell_type": "markdown",
"id": "21",
"id": "24",
"metadata": {},
"source": [
"## Visualize Clustered Heat Balance"
Expand All @@ -386,7 +441,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "22",
"id": "25",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -396,7 +451,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "23",
"id": "26",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -405,7 +460,7 @@
},
{
"cell_type": "markdown",
"id": "24",
"id": "27",
"metadata": {},
"source": [
"## API Reference\n",
Expand All @@ -432,8 +487,8 @@
"| `timesteps_per_cluster` | Timesteps in each cluster (e.g., 96 for daily at 15 min) |\n",
"| `cluster_assignments` | xr.DataArray mapping original segment to cluster ID |\n",
"| `cluster_occurrences` | How many original segments each cluster represents |\n",
"| `clustering_result` | Full [tsam_xarray ClusteringResult](https://github.com/FZJ-IEK3-VSA/tsam_xarray) |\n",
"| `aggregation_result` | Full [tsam_xarray AggregationResult](https://github.com/FZJ-IEK3-VSA/tsam_xarray) (pre-IO only) |\n",
"| `clustering_result` | Full [tsam_xarray ClusteringResult](https://github.com/FBumann/tsam_xarray) |\n",
"| `aggregation_result` | Full [tsam_xarray AggregationResult](https://github.com/FBumann/tsam_xarray) (pre-IO only) |\n",
"\n",
"### Storage Behavior\n",
"\n",
Expand All @@ -451,7 +506,7 @@
},
{
"cell_type": "markdown",
"id": "25",
"id": "28",
"metadata": {},
"source": [
"## Summary\n",
Expand All @@ -473,7 +528,7 @@
"4. **Storage handling** is configurable via `cluster_mode`\n",
"5. **Use `apply_clustering()`** to apply the same clustering to different FlowSystem variants\n",
"6. For advanced clustering options (weights, algorithms, segmentation, tuning), see\n",
" [tsam_xarray](https://github.com/FZJ-IEK3-VSA/tsam_xarray) and [tsam](https://github.com/FZJ-IEK3-VSA/tsam)\n",
" [tsam_xarray](https://github.com/FBumann/tsam_xarray) and [tsam](https://github.com/FZJ-IEK3-VSA/tsam)\n",
"\n",
"### Next Steps\n",
"\n",
Expand All @@ -497,9 +552,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
"version": "3.13.2"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
}
85 changes: 77 additions & 8 deletions docs/user-guide/migration-guide-v7.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
pip install --upgrade flixopt
```
v7.0.0 has a single breaking change: clustering now runs on
[tsam_xarray](https://github.com/FZJ-IEK3-VSA/tsam_xarray) instead of tsam.
[tsam_xarray](https://github.com/FBumann/tsam_xarray) instead of tsam.
The `transform.cluster()` call signature is unchanged — only the `Clustering`
result object and a few helpers changed.

Expand Down Expand Up @@ -80,14 +80,83 @@ tsam_xarray, so the old "non-constant inputs" preview is no longer meaningful.
See [`cluster_inputs()`](#cluster-inputs) for the v7 equivalent
(different semantics — it includes constants).

### Removed: metrics, plotting, and original-data serialization
### Changed: metrics, plotting, and original-data serialization

`Clustering.metrics` (RMSE/MAE), `clustering.plot.*`, and the
`include_original_data=...` flag on `to_netcdf()` / `to_dataset()` are gone. For
accuracy analysis or plotting, use `clustering.aggregation_result` (a tsam_xarray
`AggregationResult`) **before** serialization, or rebuild via
`Clustering.metrics` (RMSE/MAE), the `clustering.plot.heatmap()` / `.clusters()`
plots, and the `include_original_data=...` flag on `to_netcdf()` / `to_dataset()`
are gone. For accuracy analysis or plotting, use the accessors below (backed by a
tsam_xarray `AggregationResult`) **before** serialization, or rebuild via
`transform.apply_clustering(...)` after loading.

#### Comparing original vs clustered profiles

`clustering.compare()` returns a tidy `xr.Dataset` (data vars `original` and
`clustered`, on the **original** time axis) for **all** clustered variables —
select a subset on the dataset itself, e.g. `.sel(variable=...)`. flixopt bundles
the `.plotly` accessor (`xarray_plotly`), so plotting all variables at once is a
one-liner (stack `original`/`clustered` onto a `profile` dim, then facet):

```python
fs_clustered = flow_system.transform.cluster(n_clusters=8, cluster_duration='1D')
clustering = fs_clustered.clustering

(
clustering.compare() # all variables; subset via clustering.compare().sel(variable=...)
.to_dataarray(dim='profile')
.plotly.line(x='time', color='profile', facet_row='variable')
.update_yaxes(matches=None) # variables have different scales
)
```

For a single variable, `clustering.compare('HeatDemand(Q)|fixed_relative_profile')`
returns just that column.

Related accessors (all on the original time axis, with the original dim names):

| Accessor | Meaning |
|---|---|
| `clustering.original` | the input time series (dims: `variable`, `time`, plus periods/scenarios) |
| `clustering.reconstructed` | the clustered profile mapped back onto full time (same dims/order as `original`) |
| `clustering.residuals` | `original - reconstructed` |
| `clustering.accuracy` | `AccuracyMetrics` — `rmse`/`mae`/… per `variable`, plus `weighted_rmse`/… |

Available column names are `list(clustering.original['variable'].values)`
(equivalently `flow_system.transform.cluster_inputs()`).

##### Multiple variables, periods, and scenarios

These accessors keep **every** extra dimension: all time-varying inputs are
stacked on a `variable` axis, and periods/scenarios stay as their own dims. So
`clustering.original` is `(variable, time)` for a plain system and
`(period, scenario, variable, time)` with both — `clustering.compare()` mirrors
that. `accuracy.rmse` is resolved per `(variable, period, scenario)` and
`accuracy.weighted_rmse` per `(period, scenario)`; clustering runs independently
per slice. Select and facet with the natural coordinate names:

```python
cmp = clustering.compare('HeatDemand(Q)|fixed_relative_profile') # dims: (period, scenario, time)
cmp.sel(period=2030, scenario='high') # a single 1-D slice

# facet periods/scenarios with the natural coordinate names
(
cmp.to_dataarray(dim='profile')
.plotly.line(x='time', color='profile', facet_row='period', facet_col='scenario')
)
```

!!! note "Raw tsam_xarray access"
`clustering.aggregation_result` still exposes the underlying tsam_xarray
`AggregationResult` if you need it. Note it is the **raw** result, on which
flixopt's reserved-dim renames are still applied — its period dim is
`_period` (and `cluster` is `_cluster`). The `compare()` / `original` /
`reconstructed` / `residuals` / `accuracy` accessors above un-rename these
for you, so prefer them.

!!! warning "Pre-serialization only"
These accessors hold the original data and are **not** persisted by
`to_netcdf()` / `to_json()`. Access them before saving, or rebuild the result
on a freshly loaded FlowSystem with `transform.apply_clustering(...)`.

### Expansion: `disaggregate()` replaces `expand_data()` / `timestep_mapping`

```python
Expand Down Expand Up @@ -157,7 +226,7 @@ fs_clustered = flow_system.transform.cluster(
- [ ] Remove `clustering_group` / `clustering_weight` from `TimeSeriesData`; pass weights explicitly
- [ ] Replace `expand_data()` / `timestep_mapping` with `disaggregate()`
- [ ] Update removed `Clustering` properties (see table above)
- [ ] Move any metrics/plotting to `clustering.aggregation_result` before serialization
- [ ] Replace `clustering.metrics` with `clustering.accuracy`; use `clustering.compare()` (+ your plotting library) for original-vs-clustered plots (before serialization)
- [ ] Re-run `transform.cluster()` for any NetCDF saved with v6

---
Expand All @@ -166,6 +235,6 @@ fs_clustered = flow_system.transform.cluster(

- [Clustering User Guide](optimization/clustering.md)
- [Clustering Notebooks](../notebooks/08c-clustering.ipynb)
- [tsam_xarray](https://github.com/FZJ-IEK3-VSA/tsam_xarray)
- [tsam_xarray](https://github.com/FBumann/tsam_xarray)
- [CHANGELOG](https://github.com/flixOpt/flixopt/blob/main/CHANGELOG.md)
- [GitHub Issues](https://github.com/flixOpt/flixopt/issues)
Loading
Loading