protocol 1.7: outpoint subs

[SomberNight]

This defines a new Electrum Protocol version: 1.7.

changes compared to 1.6:

• the blockchain.scripthash.* methods are all replaced with blockchain.scriptpubkey.* methods
  • ref feature: blockchain.scriptpubkey.* endpoints #3
  • the new methods also contain the chaintip in their response
    • reverted in protocol 1.7: revert adding chaintip to responses #17
• new RPCs: blockchain.outpoint.subscribe RPC (and related RPCs) to subscribe to a transaction
  outpoint, and get a notification when it gets spent.
• server.ping can now also be sent as an unrequested notification, by both the client
  and the server. The request/response signature also changed.
• standardize "history too large" error code for blockchain.scriptpubkey.* methods
• new RPC: blockchain.transaction.testmempoolaccept
  • ref Add support for testmempoolaccept #11
• new RPC: mempool.recent to list a few of the latest txs just added to the mempool.
  • ref Add mempool.recent protocol method #13

---

This proposal is less ambitious than previous drafts of 1.7:

• most notably witness SPV is postponed
  • ref protocol 1.8 (WIP) #16
• the method_flavours field for the server.features response is also postponed,
  as I realised we are not clear re how to signal/negotiate optional features
  • see related Extend server.version for feature negotiation #7 idea
• blockchain.outpoint.subscribe requires spk_hint as an input param, to help server impls

---

We have a client implementation in spesmilo/electrum#10627,
and a server implementation in spesmilo/electrumx#303.

---

EDIT: note that #17 was an after-merge revert, which reverted adding the chaintip to RPC responses

[ecdsa]

LGTM

[Davidson-Souza]

Concept ACK; thank you for adding the spk_hints to blockchain.outpoint.subscribe as well.

[cculianu]

This is the first I'm seeing of this, but as the maintainer of Fulcrum I'll chime in with a brief set of points:

• removing blockchain.scripthash.* and replacing them with blockchain.scriptpubkey.* is a mistake. The scripthash thing is clever for a variety of reasons. Support both. Internally the spk one can just map to scripthash impl. anyway.
• blockchain.outpoint.subscribe -- probably a good idea
• server.ping yeah whatever. I realize this is for "privacy" but I think it's a waste of time to even worry about. This is best handled by a custom/well-engineered transport layer. Not ad-hoc imperfect solutions in the application layer. I will implement this in Fulcrum tho to be compatible, I guess.
• all the rest: yeah seems ok to me

[evoskuil]

I agree that script-hash is good to retain. Since we support all versions it’s available, so I wasn’t really thinking of it as gone. But it creates a complexity to have to negotiate different versions.

[cculianu]

I agree that script-hash is good to retain. Since we support all versions it’s available,

yeah but what if you want some stuff from v1.7 but still want to use the scripthashes.

makes no sense to remove them it’s actually superior for many use cases

mistake to remove…

[ecdsa]

I agree that script-hash is good to retain. Since we support all versions it’s available, so I wasn’t really thinking of it as gone. But it creates a complexity to have to negotiate different versions.

Version negotiation is not only about which features are available to the client, it is also about what the server can expect from the client. Light servers (such as EPS) do not index scripthashes, they can only serve requests using SPKs. Therefore it makes sense to require SPKs from the client.

[evoskuil]

Given that this is a proposed change, how could such servers been operational to date? And out of curiosity, what makes it challenging to provide an index of the hash?

[SomberNight]

@cculianu

• removing blockchain.scripthash.* and replacing them with blockchain.scriptpubkey.* is a mistake. The scripthash thing is clever for a variety of reasons. Support both. Internally the spk one can just map to scripthash impl. anyway.

For a full index server it is indeed trivial and cheap to support both. As the notifications still use scripthashes, the server would need to keep some scripthash logic, also the existing servers already have DBs that reference scripthashes. So yes, the idea is that a server impl would internally map spks to scripthashes anyway.

makes no sense to remove them it’s actually superior for many use cases

When are they superior?
I see the following advantages for scripthashes, all of them weak:

1. they are constant-length, which is good against traffic analysis
   • but then you also said for server.ping:

     I realize this is for "privacy" but I think it's a waste of time to even worry about.

   • instead we should just pad the traffic, as also described in this changeset
2. they use less bandwidth compared to long scriptpubkeys
   • but note that typical popular scriptpubkey types are of similar length:
     (from top of my head, ignoring hex, p2wpkh is 22 bytes, p2wsh is 34 bytes, p2tr 34 bytes, vs scripthashes are 32 bytes)
3. I remember kyuupichan's argument in the past was that they are good for privacy against low-skill would-be-chainalysis server operators, as if the client directly sent over their addresses, the server op would have all their addresses linked together already
   • in contrast, if the client only sends a list of scripthashes, many of which are unused, the server op needs to store them in a database for a long time and only slowly over time would the addresses get revealed when they get used
   • also if a server was blanket saving all sets of subbed unused scripthashes together, they could be spammed (tricky clients could intentionally bloat their DB)
   • while an interesting thought experiment, against a high-tech adversary such as the actual Chainalysis and the like, I believe the argument to be moot

These were the arguments we already went through internally and disregarded them. Though indeed I had not mentioned them publicly in this issue before.

@evoskuil

Given that this is a proposed change, how could such servers been operational to date? And out of curiosity, what makes it challenging to provide an index of the hash?

See discussion linked from the OP:

• the blockchain.scripthash.* methods are all replaced with blockchain.scriptpubkey.* methods
  • ref feature: blockchain.scriptpubkey.* endpoints #3

Basically light personal servers such as EPS, bwt, and Floresta can be used if you already have your own bitcoind but want to use an electrum-protocol-based wallet, and connect your wallet somehow to your bitcoind.
The way they work is that the user needs to configure the middleware with their xpub (or output script descriptors), and then

• either the middleware RPC-calls into bitcoind wallet to create a wallet using a list of imported addresses, and it is the middleware that pre-calculates the addresses and rolls ahead the gap limit
• or the middleware RPC-calls into bitcoind wallet (usually Bitcoin Core) to create a wallet using the output script descriptor
• or Floresta uses BIP-158 neutrino-style compact block filters to scan for the addresses, and those filters are indexed based on scriptpubkeys

(note that EPS and bwt are unmaintained but Floresta is very much alive)

The issue with keeping support for scripthashes is that then basically no client will migrate and Floresta and similar approaches will not be able to take advantage of the scriptpubkey subs in practice.

---

on another note, please try to keep top-level comments to a minimum: organise topics into threads, to keep it easier to follow.

[ecdsa]

note: there is an in-development EPS plugin: https://github.com/svanstaa/electrum-eps-plugin

[cculianu]

These were the arguments we already went through internally and disregarded them. Though indeed I had not mentioned them publicly in this issue before.

Yes, you tend to do that. Just unilaterally make decisions.

There is no harm in continuing to support scripthashes. Aside from the advantages you mentioned, here is another that you neglect:

• Why impose a cost on client software to update to support scriptPubKeys if it already has code written to think/talk in terms of scriptHashes?

I know it's a small update (after all to get to a scriptHash you need an spk first) -- but it's an update nonetheless! Why impose this cost at all? You can easily support both and you should.

The issue with keeping support for scripthashes is that then basically no client will migrate and Floresta and similar approaches will not be able to take advantage of the scriptpubkey subs in practice.

Huh? I said support both.

[SomberNight]

The issue with keeping support for scripthashes is that then basically no client will migrate and Floresta and similar approaches will not be able to take advantage of the scriptpubkey subs in practice.

Huh? I said support both.

Floresta is a server. I am arguing that by supporting both, clients will never migrate. This is also what you are saying by the way, you are arguing clients should not need to be changed. Except then servers like Floresta will never get a chance to be adopted, because they will never work with lazy clients.

(by lazy client I mean clients that implement the new protocol but only make the most minimal changes so e.g. would not change from sh to spk)

Clients that don't implement changes will have to keep using old protocol versions.
That provides a clear incentive structure: if a client wants to take advantage of new features, they also have to adapt their existing code for other changes.

Years from now, servers might not support old protocol versions anymore, and at that point, old clients that still will not have updated, will get broken. I think that is fine. We don't want to or need to keep accumulating technical debt.

These were the arguments we already went through internally and disregarded them. Though indeed I had not mentioned them publicly in this issue before.

Yes, you tend to do that. Just unilaterally make decisions.

As soon as someone asked, I listed the arguments.

So will you list the "actually superior for many use cases" use cases you had in mind? :)
"weak advantage" does not mean "superior"

The argument that we should keep supporting scripthashes because then clients do not have to be changed is extremely weak. By using a new protocol version, the client impl already needs to make a change! They need to signal they support it. Might as well implement some changes then. :) Or don't change anything in the client impl and keep negotiating the existing protocol versions.

Technical feedback is very welcome, that's why I tagged people. I had also tagged you. Subjective opinions are also welcome. The tradeoffs need to be weighed, but ultimately someone needs to make decisions, or there will never be any progress.

This is not like Bitcoin consensus, forking away is much cheaper. We don't need a full democracy and don't need to argue over soft forks for 5 years. Unlike in consensus rules, we can also undo changes later if they turned out to be bad decisions. I don't want the protocol to ossify, I want to be able to change things. To deprecate and remove stuff. To try new things. For example, when scripthash subs were added, soon after address subs were removed. I think that was a good change, very much worth it. I don't want the server to have to know about address encoding/decoding at all. But AFAIU you still support address subs in Fulcrum. I strongly think it's not worth it to keep supporting that.

In my eyes, one of the main points of having version negotiation is to be able to remove functionality.

In general your feedback is appreciated on these matters, as you have implemented both server and client code using the protocol, so clearly you are a subject matter expert.

[sstone]

This is fine by us (we only implement a simple electrum client), I just suggested a rewording of the description of the outpoint subscription (which will make our life easier :))

I understand that version/feature negotiation is a WIP but if script hash methods are to be removed it could be made stricter (maybe by not using strings to describe protocol versions ?)

[SomberNight]

@sstone

I understand that version/feature negotiation is a WIP but if script hash methods are to be removed it could be made stricter (maybe by not using strings to describe protocol versions ?)

What do you mean? Do you mean changing the protocol version from string to int, for easier comparison? Or do you mean replacing it with a feature vector?

[sstone]

Yes, something that is stricter and well defined (changing the version to an int, adding new explicit fields (min/max protocol) with int values, defining explicit enum values that version strings must match, ...).

[SomberNight]

adding new explicit fields (min/max protocol)

Note there is already an explicit min/max protocol that the client can send, and from that range the server selects the version to be used. (so the client can send a range, not just a single protocol version)

electrum-protocol/docs/protocol-methods.rst

Lines 1153 to 1158 in 22deba8

	* *protocol_version*
	An array ``[protocol_min, protocol_max]``, each of which is a
	string. If ``protocol_min`` and ``protocol_max`` are the same,
	they can be passed as a single string rather than as an array of
	two strings, as for the default value.

Example:

sending message b'{"jsonrpc":"2.0","method":"server.version","id":0,"params":["electrum/4.7.2",["1.4","1.6"]]}'
received data b'{"jsonrpc":"2.0","result":["ElectrumX 1.18.0","1.4.3"],"id":0}\n'

One shortcoming however is that the client must support "all" protocol versions in the range it sends :/
(there is some discussion in #7)
Perhaps in the future the client could send an explicit list of versions in supports - but to make it backwards compatible, it could maybe send this in addition to the min/max range (as long as the client wants to be compatible with older servers).

As for simply changing the version string to a version int, I don't think that would gain us much: it is already fairly simple to compare the string versions. (also, we can't naively change the protocol version strings to ints, as that would be a major breaking change in the version negotiation)

but yes, I see there is a lot of room for improvement...

[ecdsa]

ACK 5a702d4

[SomberNight]

Thanks for the reviews and feedback.

[SomberNight]

Argh... I think we should revert including the chaintip in responses (9c32658), and postpone that for later. (Or rather, do something similar, but not exactly that, in the next version)

Sorry... I should have noticed the issue before merging this. :(

---

I just realised the implementation in ElectrumX (server) was not including the chaintip in the notifications for "blockchain.outpoint.subscribe". (I missed this previously as the integration tests between Electrum Wallet and ElectrumX were passing: the client saw the missing field, logged an error and disconnected from the server, but then the client reconnected, re-subscribed to the outpoints, and got responses from the server with the chaintip included - as only the notifications were missing the chaintip, the responses were not. As the reconnect self-healed the state, everything the tests are actually checking passed.)

Turns out with ElectrumX's architecture and how it is using bitcoind, it is not simple at all to also include the chaintip in the notifications... "blockchain.outpoint.subscribe" is implemented using bitcoin core's txospenderindex. ElectrumX calls "getrawtransaction" to find the funder tx, and then it calls "gettxspendingprevout" to find the spender tx, and for both responses bitcoind gives the containing blockhash for the tx - but not the chaintip. After that, ElectrumX uses its own mempool to enrich the "blockchain.outpoint.subscribe" response with the mempool state of funder tx and spender tx. So the server needs to make sure bitcoind's chaintip does not move and its own mempool stays consistent during all this. This is doable when serving the initial request: ElectrumX checked the chaintip before doing all the calculations and then checked the chaintip again, and if it changed, it tried again, which was already hackish. In the worst case, the server would fall back to sending a json-rpc-error to the client. but this is not practical at all while sending out notifications.

In general the non-monolithic architecture between bitcoind and the electrum server middleware is making committing to a chaintip difficult. For example, consider the bitcoin core "getrawtransaction" RPC: if it does not find a tx at all, how can the server commit to a chaintip? The response from bitcoind does not contain it. The server would have to get bitcoind's tip before and after calling "getrawtransaction", and compare them, and then retry the whole thing if the tip moved. (which is effectively already what I was doing in ElectrumX when serving the request, which is already ugly, but it would be even worse to do the same when sending out notifications...)

I am assuming that other server implementations would eventually run into a similar issue.

---

Also, note that when serving the history of an spk or the status of an outpoint, if all the txs touching it are somewhat-deeply mined, it is almost of no interest if the chaintip moves. Even a small reorg is not interesting if it does not affect the relevant txs as they are deeper.

So, now I think the right approach is that instead of adding the chaintip to responses, we could add the blockhash corresponding to the last relevant tx.

• e.g. for spk.get_history, the last tx touching the spk,
• or for outpoint.get_status, the spender if it exists, or the funder if that exists, or none
  • also note that if a server implements this using bitcoind's txospenderindex, "getrawtransaction" and "gettxspendingprevout" already provide the relevant blockhash
• if there is no relevant tx, e.g. the funding tx for an outpoint is not found, or the history of an spk is empty, we would not include any blockhash (as opposed to including the chaintip). Including the chaintip is just too fragile as it can change at any time.

However, I think this needs more discussion.

---

So the simplest approach is to revert just this small part of 1.7 for now, remove it from 1.7, move forward with the rest of 1.7, and see how to include something along these lines in the next version.

see #17

---

@tnull as you asked for this featured originally, could you perhaps open an issue describing exactly what you would like (how you would want to use this), so we can discuss there what alternative we could do instead. e.g. would what I just sketched out above work too?

[tnull]

@tnull as you asked for this featured originally, could you perhaps open an issue describing exactly what you would like (how you would want to use this), so we can discuss there what alternative we could do instead. e.g. would what I just sketched out above work too?

Yeah, no worries regarding the revert if it otherwise would block the release. I may open an issue at some point, but it sounds that it would be preferable if similar API changes happen on Bitcoin Core itself, to make actual implementations of this easier?

[ecdsa]

What I don't like about sending the chain tip is that it is not directly related to an address or outpoint history. We want a new notification to be sent if the address or outpoint history is affected by a reorg. Thus, it makes more sense to include the block hash of the most recent element of the history in the response, because that is the change that triggered the new notification. We could even include this hash in the address status, so the client would actually see a new status if the tx block hash has changed.