Fix cifar vulnerability#205
Merged
vijay-arya merged 13 commits intoJul 23, 2026
Merged
Conversation
Collaborator
|
@monindersingh could you pls make this minor change to avoid the DCO failure. |
Signed-off-by: Moninder Singh <moninder@us.ibm.com>
Security fix for aix360/datasets/cifar_dataset.py (HIGH, 7.8)
CWE-22 (path traversal / tar-slip) + CWE-502 (untrusted deserialization)
Root cause
----------
_download_data() fetched cifar-10-python.tar.gz over the network and then:
- called tar.extractall() with no member validation, allowing a malicious
or MITM'd archive to write files outside the target directory (tar-slip),
and
- _process_data() loaded the extracted batch files with pickle.load(), which
executes arbitrary code if the archive was tampered with.
Both risks share one root cause: the downloaded archive was never verified to
be authentic before its contents were trusted.
Changes
-------
1. Archive integrity verification (CWE-502)
- Added CIFAR10_ARCHIVE_SHA256, pinned to the digest of the authentic
archive. The pinned value was computed from a downloaded copy only after
verifying it against the vendor-published MD5
(c58f30108f718f92721af3b95e74349a).
- _download_data() now hashes the downloaded archive (_sha256_of_file) and
compares it to the pin BEFORE any extraction or pickle load. On mismatch
it deletes the archive and raises (fail-closed). Because the pickle format
is intrinsic to the CIFAR-10 release, provenance verification is the
appropriate mitigation.
2. Safe tar extraction (CWE-22)
- Added _safe_extract(), which validates every member resolves inside the
target directory and rejects symlink/hardlink members whose targets escape
it, then extracts using the tarfile 'data' filter (Python 3.12+) with a
fallback for older Python versions.
- Added _is_within_directory(), which compares on a path-separator boundary
(not a raw string prefix) so a sibling such as /a/bad is not treated as
being inside /a/b.
- On any extraction failure, partially-extracted contents and the archive
are removed before re-raising.
3. scikit-learn compatibility (not security-related; same file)
- OneHotEncoder(sparse=False) fails on scikit-learn >= 1.2, where the
argument was renamed to sparse_output. Routed both call sites through a
version-agnostic _onehot_dense() helper so the loader runs across
scikit-learn versions.
Verification
------------
Exercised end-to-end against the genuine archive:
- authentic archive verifies, extracts, and processes the full pipeline;
produces all six JSON outputs with correct shapes ((30000,32,32,3) and
(30000,10)) and cleans up the archive and batch directory afterward;
- a tampered archive is rejected with a clear SHA-256 mismatch error and the
bad file is removed;
- a tar member with a traversal path is rejected and no file is written
outside the target directory.
Files changed
-------------
aix360/datasets/cifar_dataset.py
Signed-off-by: Moninder Singh <moninder@us.ibm.com>
Replace retired ubuntu-20.04 with ubuntu-latest, upgrade actions/checkout from v3 to v4, and actions/setup-python from v4 to v5 to resolve runner timeouts and Node.js 20 deprecation warnings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Dennis Wei <dwei@us.ibm.com> Signed-off-by: Moninder Singh <moninder@us.ibm.com>
cvxpy 1.4+ no longer bundles the ECOS solver, causing the rbm tests to fail with "The solver ECOS is not installed." Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Dennis Wei <dwei@us.ibm.com> Signed-off-by: Moninder Singh <moninder@us.ibm.com>
On arm64 macOS runners, Homebrew installs to /opt/homebrew which is not in the default compiler search path. Export CFLAGS/LDFLAGS so pygraphviz can find the graphviz headers and libraries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Dennis Wei <dwei@us.ibm.com> Signed-off-by: Moninder Singh <moninder@us.ibm.com>
rule_induction, tsice, tssaliency, and tslime pin pandas<2.0.0 but left numpy unconstrained. pip installs numpy 2.x which is binary- incompatible with pandas 1.x, causing "numpy.dtype size changed" errors at import time. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Dennis Wei <dwei@us.ibm.com> Signed-off-by: Moninder Singh <moninder@us.ibm.com>
Signed-off-by: Giridhar Ganapavarapu <giridhar.ganapavarapu@ibm.com> Signed-off-by: Moninder Singh <moninder@us.ibm.com>
…nloader
Addresses two security advisories against the CEM_MAF explainer:
* CVSS 9.2 (CWE-502) — pickle.load + exec(build_module_src) RCE in the
Karras PGG celebA-HQ GAN loader.
* CVSS 8.5 (CWE-22 + CWE-502) — path-traversal / pickle deserialization
in the unreachable load_AE helper.
Changes:
- GAN loader (CEM_MAF_aen_PN.py): replace pickle.load + exec of the
embedded build_module_src with a safetensors weights file plus a JSON
static-kwargs sidecar. The trusted G_paper build function still comes
from progressive_growing_of_gans/networks.py; no untrusted source is
exec'd at runtime.
- load_AE (CEM_MAF_utils.py): deleted. No caller reachable in-tree.
- Downloader (dwnld_CEM_MAF_celebA.py): rewrite to fetch from two
Hugging Face repos over HTTPS with SHA-256 verification. 78 files
pinned; every download streams to <dest>.part, hashes on the fly, and
is os.replace()-moved into place only on match. Unknown names are
rejected. Path traversal is impossible: filenames come from the
caller-provided list, dest paths use os.path.join with basename-only
components.
- attr_classifier_device / gan_device kwargs (CEM_MAF.py,
CEM_MAF_aen_PN.py, CEM_MAF_aen_PP.py): pin the Karras GAN and the
per-attribute classifiers to /cpu:0 on Ampere/Hopper to dodge a
TF1.15 + bundled-cuBLAS-10 SGEMM bug that NaN-corrupts / silently
non-determinises the forward pass on A100/H100. Default None
preserves pre-existing behavior on pre-Ampere GPUs.
- Notebook (CEM-MAF-CelebA.ipynb): remove manual Google-Drive-pickle
setup instructions; fetch the GAN artifacts via dwnld_celebA_gan()
alongside the other automatic downloads; pass '/cpu:0' for both
device knobs.
- .gitignore: exclude runtime-fetched model + data files (they now come
from Hugging Face on demand) and other local-only artifacts.
Verified end-to-end on a fresh working tree with all files re-downloaded
from Hugging Face: notebook produces Added High_Cheekbones (class 4→6)
for image 15, matching pre-rewrite results bit-for-bit.
Signed-off-by: Ronny Luss <rluss@us.ibm.com>
Signed-off-by: Moninder Singh <moninder@us.ibm.com>
The Karras et al. 2018 progressive-growing GAN weights bundled via Hugging Face are released under CC-BY-NC 4.0, distinct from AIX360's Apache-2.0 license. Add the full CC-BY-NC 4.0 text under `supplementary license/`, note the attribution in the licenses README, and mention it in the CEM-MAF-CelebA notebook where users encounter the GAN. Signed-off-by: Ronny Luss <rluss@us.ibm.com> Signed-off-by: Moninder Singh <moninder@us.ibm.com>
The CEM_MAF vulnerability fix (d072a36) replaced pickle.load of the Karras GAN with safetensors, but did not add safetensors to setup.py's contrastive extras. Pin <0.4 so Py3.6 wheels remain available under the tensorflow==1.14 / keras==2.3.1 environment. Signed-off-by: Ronny Luss <rluss@us.ibm.com> Signed-off-by: Moninder Singh <moninder@us.ibm.com>
The CEM_MAF vulnerability fix (d072a36) .gitignore'd the whole progressive_growing_of_gans/ shadow directory as "Karras PGG upstream (Google Drive pickle, no longer used at runtime)". That was too broad: CEM_MAF_aen_PN.py top-level imports G_paper from networks.py inside that directory, so a fresh clone would ImportError. Copy the code CEM_MAF actually needs -- G_paper plus the twelve helpers it transitively calls -- into a new gan_networks.py sibling module. Swap the import to match. The rest of the upstream directory (D_paper discriminator, NVIDIA training scripts, sample images, .pkl weights) stays untracked; those are neither imported nor safe to distribute. The vendored code is licensed CC-BY-NC 4.0; attribution and modification notes are in the file header, and the full license text was already committed under supplementary license/LICENSE-CC.txt in 5b286a7. Signed-off-by: Ronny Luss <rluss@us.ibm.com> Signed-off-by: Moninder Singh <moninder@us.ibm.com>
The safetensors dependency added in 15a1e26 has no Py3.6 wheels and its sdist requires PyO3 >= 3.7, breaking pip install .[contrastive] on the CEM CI job. Bump the job to Python 3.7: - .github/workflows/Build.yml: rename build-cem-on-py36 to build-cem-on-py37 and set python-version to ["3.7"]. - README.md: update the contrastive row's Python column to 3.7. profwt and shap rows keep 3.6 -- their extras don't include safetensors, so nothing forces a bump there. Also pin protobuf<3.20 in the contrastive extras. On Py3.6 pip happened to resolve to protobuf 3.19.x (last cp36 wheel) which TF 1.14 tolerates; on Py3.7 pip resolves to protobuf 4.x, whose descriptor API refactor makes TF 1.14's generated _pb2.py files refuse to import. The pin makes the resolution explicit rather than accidental. Correct the safetensors comment while we're here: the <0.4 bound is about 0.3.x being the last series with Py3.7 wheels, not about Py3.6. Signed-off-by: Ronny Luss <rluss@us.ibm.com> Signed-off-by: Moninder Singh <moninder@us.ibm.com>
http://yann.lecun.com/exdb/mnist/ has returned 404 for the raw .gz files since ~2022, silently breaking any test or notebook that constructs MNISTDataset() without pre-populated files. Surfaced on the CEM CI job now that its install path resolves cleanly and the test actually runs to the download step. Switch to https://ossci-datasets.s3.amazonaws.com/mnist/, the S3 bucket the PyTorch team stood up as the community successor mirror after LeCun's copy died. Serves byte-identical files at the same paths; also upgrades the transport from http to https. Signed-off-by: Ronny Luss <rluss@us.ibm.com> Signed-off-by: Moninder Singh <moninder@us.ibm.com>
monindersingh
force-pushed
the
fix-cifar-vulnerability
branch
from
July 23, 2026 13:25
9b6a12b to
db2da01
Compare
Contributor
Author
|
@vijay-arya I tried to follow the instructions and the error is gone. But now my PR shows 13 commits, 20 changed files....don't know if that is correct? I synched my base with main before following the instructions to avoid the DCO failure. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix tar-slip and pickle deserialization vulnerabilities in CIFAR loader
Security fix for aix360/datasets/cifar_dataset.py (HIGH, 7.8)
CWE-22 (path traversal / tar-slip) + CWE-502 (untrusted deserialization)
Root cause
_download_data() fetched cifar-10-python.tar.gz over the network and then:
or MITM'd archive to write files outside the target directory (tar-slip),
and
executes arbitrary code if the archive was tampered with.
Both risks share one root cause: the downloaded archive was never verified to
be authentic before its contents were trusted.
Changes
Archive integrity verification (CWE-502)
archive. The pinned value was computed from a downloaded copy only after
verifying it against the vendor-published MD5
(c58f30108f718f92721af3b95e74349a).
compares it to the pin BEFORE any extraction or pickle load. On mismatch
it deletes the archive and raises (fail-closed). Because the pickle format
is intrinsic to the CIFAR-10 release, provenance verification is the
appropriate mitigation.
Safe tar extraction (CWE-22)
target directory and rejects symlink/hardlink members whose targets escape
it, then extracts using the tarfile 'data' filter (Python 3.12+) with a
fallback for older Python versions.
(not a raw string prefix) so a sibling such as /a/bad is not treated as
being inside /a/b.
are removed before re-raising.
scikit-learn compatibility (not security-related; same file)
argument was renamed to sparse_output. Routed both call sites through a
version-agnostic _onehot_dense() helper so the loader runs across
scikit-learn versions.
Verification
Exercised end-to-end against the genuine archive:
produces all six JSON outputs with correct shapes ((30000,32,32,3) and
(30000,10)) and cleans up the archive and batch directory afterward;
bad file is removed;
outside the target directory.
Files changed
aix360/datasets/cifar_dataset.py