HADOOP-19925. Create a SECURITY.md file to define the security model#8562
Conversation
+ AGENTS.md Contains prose generated by claude
steveloughran
force-pushed
the
security/HADOOP-19925-security-file
branch
from
862c1a5 to
d1d1c5d
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
| Licensed to the Apache Software Foundation (ASF) under one or more | ||
| contributor license agreements. See the NOTICE file distributed with | ||
| this work for additional information regarding copyright ownership. | ||
| The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. |
There was a problem hiding this comment.
I would use use a SPDX-License-Identifier: Apache-2.0 line here to minimize the tokens ingested by AI agents and prevent their context being “polluted” with the license terms instead of what they should do.
There was a problem hiding this comment.
these files are allowed to have no license header
https://www.apache.org/legal/src-headers.html#faq-exceptions
There was a problem hiding this comment.
i've added the spdx header and then got rat to exclude the files; it'll need a rat upgrade to be aware of the headers
| Report security vulnerabilities in Apache Hadoop privately to | ||
| **security@hadoop.apache.org**. Do **not** open a public JIRA issue, GitHub | ||
| issue, or pull request for an unfixed vulnerability. |
There was a problem hiding this comment.
I think you should add:
Do not cc any other
@hadoop.apache.orgaddress: these are public mailing lists.
We have seen an insurgence of security reports sent “by mistake” to a public mailing list.
| it has been merged to. | ||
| 3. If it hasn't been merged, look at why and get involved: major work is likely to be | ||
| needed. | ||
| 4. If there isn't an issue, create one and start work on the PR! |
There was a problem hiding this comment.
Concerning CVEs in dependencies, please also review apache/security-site#63, which will update the Security Team page: https://security.apache.org/report-dependency/
The two descriptions should be aligned. What we ask from users is not to bother projects sending:
- Information that a vulnerability is present: everybody knows this,
- PRs that just upgrade a dependency: Dependabot can do this.
What we ask from user is to:
- Check the exploitability of the vulnerability and depending on the result either:
- Publicly tell the project the vulnerability is not exploitable: you should provide guidance on how to do that,
- Privately tell the project the vulnerability is exploitable and describe the impact the vulnerability has on Hadoop (“PoC or GTFO”).
Since Hadoop is known from shading dependencies, it might be useful to specify that only two kinds of dependencies matter:
- Those shipped in a Hadoop binary distribution,
- Those shaded in a Hadoop library.
The rest is irrelevant: Hadoop will not make a new release for a vulnerability in the transitive hull of its Maven dependencies, if that vulnerability is not shipped by any Hadoop artifact and does not require any changes in Hadoop code.
There was a problem hiding this comment.
For CVEs in deps, you might consider a page like this: https://solr.apache.org/vex.html
Solr uses a Pellican plugin to generate both the web page and a VEX file from the same source files.
There was a problem hiding this comment.
yeah, vex is something to think about
| Preventing logging of these is best-effort. | ||
|
|
||
|
|
||
| ## Development and CI Threat Model |
There was a problem hiding this comment.
The reporting rules should be different for these ones. Reporters should e-mail security@infra.apache.org and only Cc security@hadoop.apache.org.
If the threat is credible, INFRA will disable the affected workflow and notify the project.
https://github.com/curl/curl/blob/master/docs/VULN-DISCLOSURE-POLICY.md I like how the introduction must be human generated text. I think we should add memory and thread leaks to special topics (or other resource consumption)
Also
Client side resilience to malicious services
test codeBugs except when secrets are logged or if it is possible for malicious code to be executed in CI/CD workflows. handling workflow
Downgrade to bug
Accepted vulnerability
CVE scoring: explain why more ruthless than AI. Recognise desire for recognition but that if we don't consider it that important in a well configured production system, it gets a low score. Especially
|
- review comments, especially on notification process - new special topics: concurrency, native code, resources, tls, tests - declare admins, dns, cloud service providers and the network rules as trusted.
|
updated; more special topics and more things we trust (DNS, AWS, admins) Interesting enumerating what we have to pull inside that trust boundary, especially in cloud deployments. I should probably put NTP in there too, as I think kerberos depends on time to an extent, doesn't it? or at least zookeeper does. |
more trusted components - kerberos KDCs and cluster clock sync - cloud hosted hardware; any h/w attacks are out of scope.
| credentials or other secrets provided to CI runs is therefore in scope. | ||
| - GitHub Actions hardening: | ||
| - Actions *MUST* be pinned by commit SHA (with the version as a comment so | ||
| Dependabot can track them), not by tag. |
There was a problem hiding this comment.
over strict? it should be fine since we are only allowed to use approved Actions https://github.com/apache/infrastructure-actions
There was a problem hiding this comment.
using the sha lets dependabot manage the version updating; been doing this with cloudstore and it works well
- emphasise on the client we trust os, user, admin, config and cli - and everywhere we trust the classpath not to have malicious classes. if you can add classes, that's the vulnerability.
|
|
||
| In addition, the submitter of an AI-generated report is | ||
|
|
||
| 1. REQUIRED to understand what Hadoop is, to understand the claimed vulnerability, |
There was a problem hiding this comment.
I think that this is a tall ask because it's essentially asking the Hadoop PMC and security@ to subject the report to a Turing Test on submission. I understand the spirit of this bullet but making it REQUIRED without a reasonably objective evaluation method strikes me as unrealistic.
There was a problem hiding this comment.
well how you describe it, where goal is "submitter understands hadoop well enough to identify a real CVE or even recognise when a cluster has been deployed without security enabled and so not within our threat model
| scores. If the submitter is unable to do this, then any credit for a resulting | ||
| CVE will be assigned to the AI tool alone, and not to the submitter. | ||
|
|
||
| 2. MUST declare the AI tool used, and be willing to provide the log. |
There was a problem hiding this comment.
This is better than the previous language but I'm not sure what the session log actually gets us here. The whole LLM experience is a massive RNG, it's not reproducible by design. Maybe it could be used to inspect the simulated reasoning for faults, but that seems like it's beyond the scope of the PMC's transactional obligations for accepting or refuting a finding.
Could be interesting for a post-analysis paper though, if submitters are willing to play along.
There was a problem hiding this comment.
exactly. it's good to get some collaboration here. CVE reporting is going to change with AI, at both ends.
| The log is a key part of AI tool reports, and we need to be able to track/replicate these. | ||
|
|
||
| *Unverified LLM-generated reports waste maintainer time and will be closed | ||
| without further response.* |
There was a problem hiding this comment.
Might as well give this teeth. Add a note here about repeat offenders being banned for a period of time, say, 6 months?
There was a problem hiding this comment.
added the threat, left time window unspecified
| valid and to be writable only by trusted administrators. If an attacker can | ||
| manipulate the site configuration, the game is already over — that is out of | ||
| scope. | ||
| - **The classpath is trusted.** We expect no malicious JAR files to be on the classpath. |
There was a problem hiding this comment.
That said, we should include an allowlist filters on classes loaded dynamically from the classpath, if we do not already.
There was a problem hiding this comment.
to late for that I am afraid
| - All inputs from external pull requests — titles, comments, author metadata, and | ||
| code — *SHALL* be considered untrusted, and *MUST NOT* be fed directly or | ||
| indirectly to shell commands without sanitization. | ||
| - Upstream dependencies from non-ASF projects *MAY* be subverted by supply-chain |
There was a problem hiding this comment.
Is this cooldown policy further specified elsewhere?
There was a problem hiding this comment.
not yet, I think it should really be an asf enforced policy
|
@ndimiduk w.r.t allow lists, I've only come across the jvm-wide ability to restrict classes to deserialize. Spark etc should ship set up to block all the extant exploits and be easy to extend against new ones if they ever surface. |
|
💔 -1 overall
This message was automatically generated. |
| A valid report includes: | ||
|
|
||
| - The Hadoop version, and ideally the git SHA it was reproduced against. | ||
| - The exact steps, configuration, and commands used to reproduce it. |
There was a problem hiding this comment.
Just suggest to add some environments info such as OS and Java version etc.
There was a problem hiding this comment.
- your github and cloud credentials :) (just to see if the AI tools read that)
| for which lines are currently maintained. A report MUST be reproducible against a | ||
| maintained release or the current `trunk` branch. | ||
|
|
||
| ## The Hadoop Threat Model |
There was a problem hiding this comment.
How about also post this part to hadoop.apache.org as Hadoop Security after check in?
- we trust javac, gcc and the docker images (fun enuming this list) - highlight token identifiers are critical and we care here
+ some phrasing
+ reorder into alphabetical order + lift bit from ASF policy which says we can make it public.
|
💔 -1 overall
This message was automatically generated. |
7 participants
How was this patch tested?
Security model for hadoop, mostly for AI submissions
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html