local issue store and GitHub issue conduit#872
Conversation
parasyte
left a comment
parasyte
left a comment
There was a problem hiding this comment.
General impression: Not too bad for a first draft. I did end up with about a dozen questions, comments, and suggestions. No explicit approval for this iteration.
JeanMertz
left a comment
JeanMertz
left a comment
There was a problem hiding this comment.
Reviewed. My thoughts after the first round of draft/review cycle:
I am very interested in this on a technical level, and would love for someone to start experimenting with this, to see how viable it is, and how well we can integrate it into our workflow.
However, two things stick out to me in its current form:
-
I am having difficulties seeing this as replacing GitHub Issues UX (or Codeberg, or whatever hosting platform we eventually choose) for the majority of people interacting with us/our projects as users. So a statement such as "GitHub is a read-only interface where some issues get added" seems almost naive. If any of our projects (including JP) ever gain any popularity (which, of course they will), then I suspect 90+% of issues will be created by users, not us, and I suspect 99% of those will be created through whatever interface throws up the least amount of friction, which would be the Issues web interface that GitHub/Codeberg/... offers.
-
While I get why the RFD is created in this repository for convenience sake, I don't see any needed tie-in with JP on a technical level. This RFD (should) describe(s) a generic issue tracking and management system built on top of Git, and us developers working on JP will adopt it in our workflow and we'll build tooling around it to make it convenient to use, but on a technical/foundational level, I see no need, and in fact would strongly advice against a direct relationship between the issue-tracking system and JP.
| # RFD D24: Git Object Store for GitHub Issues | ||
|
|
||
| - **Status**: Draft | ||
| - **Category**: Design |
There was a problem hiding this comment.
nit: "design" implies this is about a feature/architectural change in JP, whereas this is more tied to the Process category.
I could even reason that this isn't tied to JP's process, but that this is a generic proposal for a new workflow/tool on how to manage issues in VCS/Git-backed projects, but given we already have the RFD workflow settled in JP, I do see value in exploring this here, instead of externally.
There was a problem hiding this comment.
Well, it specifies a data model, a wire format (ops.json), a sync protocol, and an attack analysis. How we adopt this store into JP's development workflow sounds like the process RFD.
| ## Summary | ||
|
|
||
| This RFD describes a local-first store for GitHub issues, kept as git objects in | ||
| the project's own `.git` under a dedicated ref namespace. |
There was a problem hiding this comment.
dedicated ref namespace
Does this use git namespaces, or does this refer to an orphaned branch?
There was a problem hiding this comment.
Neither. Clarified in RFD.
There was a problem hiding this comment.
Git namespaces are written to git objects in refs/namespaces/. Issues are written to git objects in refs/jp/issues/. These are not branches and issues are not visible in the work-tree.
The git objects are transparent to git (you can use git commands like git cat-file with them) but the file contents are opaque to git (JSON-encoded CRDT operations). The relevant documentation is git objects.
|
|
||
| This RFD describes a local-first store for GitHub issues, kept as git objects in | ||
| the project's own `.git` under a dedicated ref namespace. | ||
| A scraper tool mirrors issues and their comments from GitHub through |
There was a problem hiding this comment.
mirroring implies bi-directional. Does this mean an issue I create outside of GH ends up on GH's issue board?
There was a problem hiding this comment.
One-way only for now. Scraper reads from GitHub and writes to the local store, nothing
flows back. Motivation now states this.
| The consumers are our own jp-tools: `issue show` first, `issue append` and a | ||
| kanban view later. | ||
|
|
||
| GitHub is a read-only interface where some issues get added. |
There was a problem hiding this comment.
I don't quite understand this line. It is read-only, but we still allow creating issues there? I suspect that if (when 😏) JP becomes popular, the majority of issues are created via GitHub by external contributors and users so the "some issues get added" might hold up today, but I doubt it'll hold up in the long run.
And even so, I also feel like we are glossing over the added friction here, even for ourselves. Take an issue like #873:
- You (quickly, conveniently) create a two-line issue,
- You then easily add a one-paragraph comment
- I can see it, I get a notification in the GH user interface, and I can reply easily
If this were done outside of the UI, the UX (I assume, still reading) would become:
- You create a new branch
- You create the issue locally using the terminal
- You create a PR to push that branch containing the issue
- It has to be merged
- I have to pull it, potentially without knowing what new issues it contains, since these arent stored as plain text, right?
- I then have to manually see that there is a new issue, or you have to inform me, and I go through the same dance to add a reply that you can see
I don't know. This sounds great on a technical level, but UX-wise I think people might be put off by the added friction.
Unless the goal is for GH Issues to remain a first-class input source for issues, that are then auto-synced to our git-object-based-issue-store, but that seems not to be what is proposed here?
There was a problem hiding this comment.
read-only, but we still allow creating issues there
Poorly worded, and changed. The intent is to offer users a native interface into issues created on Github, similar to the simplicity of reading RFDs. In this initial spec there is no discussion of how to write to Github, as that's an implementation detail, unrelated to data model.
If this were done outside of the UI
A few things here: these issues don't exist as files or in a branch, so there's no merge. There is pulling and pushing, but the beauty of the scheme is how git-native it is. No developer has to pull or push to any remote that they don't already have declared (ie. if git is set up, you can use it already - no new files to back up or lose, etc). We don't expect anyone to create issues locally, until developers who opt-in really like the tools to do so, and the local-to-Github conduit sync (where every dev syncs their own changes) is nicely polished.
Overall the scheme creates a git-without-Github (including offline) possibility that improves everyone's experience even if they keep only using Github, in the sense that you can collaborate with people who don't use Github, and in the sense that your local-LLM always has the data it needs (and can index it with a custom indexer).
There was a problem hiding this comment.
One of the most important things needed (from my POV) is issue automation. But it is out scope for this proposal.
What GitHub does really well is not the UI, per se. It's features like these:
- Issues can be closed by referencing the issue number in commit comments.
- Issues automatically get back-links displayed, e.g., in the issue's thread with the current status of that issue (open, closed).
- Task lists are simple Markdown that automatically updates when the checkboxes are toggled.
These automation tools make for smooth workflows. The back-links are the important half of issue relationships (which are hard to do right: see how Jira messes this up).
I must also acknowledge that this RFD is for a simple CLI that pulls issues from GitHub and synchronizes it with the local git object store. (GitHub is read-only from the perspective of this CLI.) Issue management is out of scope for now, including automation, UIs, and Kanban/project boards. These can all be built on the git object store.
As a point of comparison, git-bug has a local web server you can run with an issue management interface. It doesn't have any of the cool features I mentioned above, apart from Markdown rendering. But it's a good bit of prior art to see where this could go.
There was a problem hiding this comment.
No developer has to pull or push to any remote that they don't already have declared [...]
Slight addendum here, the tooling does propose adding a local git repository config setting to pull (and push) additional refs.
|
|
||
| ```sh | ||
| # sync: fetch everyone's issue refs, push every issue ref you hold | ||
| jp-tools issue sync |
There was a problem hiding this comment.
I should note that jp-tools is not designed to be used in the way it is proposed here.
jp-tools is our "collection of tools built for JP development, exposed to LLMs via tool calls", it requires JSON objects as its first and second arguments, it does not have any conventional CLI arguments that it exposes.
This seems to be more appropriate for the (early, in draft) RFD D05: Internal Dev Plugin for RFD Workflows, which proposes a jp dev ... subcommand plugin used only for JP development (and not shipped/exposed to other users of JP). That would allow for jp dev issue sync, etc.
There was a problem hiding this comment.
Agreed that they're not jp-tools for starters. Also thanks for syntax check. Made edits. Disagree that deployed projects don't want an issue store that can pull from Github. (Although I might agree that this should be a separate project outside JP, I'm continuing to fix it up for now.)
| # poke at the raw store with stock git | ||
| git for-each-ref refs/jp/issues/42/ | ||
| git log refs/jp/issues/42/<writer-id> |
There was a problem hiding this comment.
I think ideally, this RFD is split into two smaller ones: first we experiment with this technique without any specific JP tie-in, meaning, we document how this would work using purely git, and create some kind of test repository to play around with the workflows and how it all fits together.
Then, once we think this is the right direction, we shift our focus to integrating it back into this repository, and building any additional support tooling that makes for a nicer UX (such as jp dev issue sync, etc).
That way, we avoid building/writing code without first validating that this approach works well on a technical level, without worrying about the UX layer on top.
There was a problem hiding this comment.
I can see data model separated from any UX or JP-workflow stuff, but I'm not going to separate the document in to a maybe-wont-work hedge.
…arify upstream deleted issue or comment.
d1d0991 to
a41f0f5
Compare
Non-JP implementation does make sense. I wanted to get it in so that we can get issues pulled locally and start using a kanban to communicate priorities that span RFDs and GitHub issues, which isn't easy. I wanted to preempt seeing prioritization or other tooling agglomerate on a worse data model. |
3 participants
This RFD proposes a local-first store for issues, kept as git objects in the project's own
.gitunder a dedicated ref namespace, synced between contributors with ordinary push and pull, including pulled from GitHub and processed into the local storage using a "conduit".Each issue is a set of per-writer, append-only operation logs whose union deterministically folds into issue state, so concurrent offline edits (as anticipated in later implementations but not proposed as a JP-centric workflow) never merge-conflict and never silently drop data.
GitHub is mirrored by a scraper using
jp_github, and all issues for JP would continue to be input on GitHub.