Transcript for #bitcoin-dev 2018/01/31

05:49 d3vt4r sup hommies
05:49 d3vt4r Prices are crashing came here for refuge
10:53 snoogle hi
11:12 wumpus ageis: getchaintxstats has a field 'txcount' that is the total number of transactions
16:41 gielbier_ hmz. 0.15.99 validateaddress tb1qw508d6qejxtdg4y5r3zarvary0c5xw7kxpjzsx gives true. 0.16.0rc1 false . Bug?
16:42 contrapumpkin #bitcoin-core-dev might be able to give you a better answer
17:02 arubi gielbier_, are you on testnet on both?
17:19 gielbier_ regtest
17:19 gielbier_ but yeah on both
17:23 arubi gielbier_, right, regtest bech32 start with bcrt1
17:23 arubi so tb1 is not a valid prefix on regtest (finally!) :)
17:24 arubi so maybe that wallet was created with testnet? or core used to output testnet bech32 addresses for regtest?
17:25 arubi example of regtest bech32 : bcrt1qq0a6yzzjdfutgt45x38cztj4xyzwwvjnle0xat
17:28 gielbier_ ah ok :)
17:28 gielbier_ yeah it was an old testnet wallet i think
17:37 gielbier_ but we used unit test for testnet/regtest . needed to update them for bech32
17:39 arubi nice, the more bech32 the better
21:17 reardencode I'm working on a BIP to enable transaction inputs to expire if they aren't mined by an MTP/height, anyone interested in discussing / offering an early review?
21:17 arubi is that supposed to be consensus or just policy?
21:17 reardencode consensus
21:18 arubi soft fork? how then?
21:18 reardencode yes, soft fork, similar to BIP68, adding additional validation to nSequence
21:19 arubi it's a bit of an issue to put that in script
21:20 reardencode not talking about adding it to OP_CHECKSEQUENCEVERIFY yet - no script behavior changes at this time
21:20 arubi so how are you going to enforce it?
21:21 reardencode if an input with sequence "don't mine me after MTP 2018-01-31" is mined into a block with MTP 2018-02-01, nodes reject the block
21:22 reardencode (this is the same as how BIP68 is enforced on nSequence, only in the opposite direction)
21:24 arubi yes, I understand that, there's an issue with that thinking. (brb)
21:25 reardencode awesome, I'd love to hear the issue!
21:28 arubi back. reardencode, in bitcoin there's a property of transaction validity that says that once a transaction is valid, it can't become invalid. it makes managing mempool and block reorgs a lot simpler
21:29 reardencode arubi: ah, I didn't know that! can you link me to some resources on that topic?
21:30 arubi with this change you'll have to constantly chcekc and evict transactions from your mempool as they decay in time, and also it makes payments with these transactions a bit worse which ruins fungibility
21:30 arubi oh I don't know if it's documented anywhere :)
21:31 reardencode lol - trying to think this through from a software engineering perspective: I don't think you'd need to keep scanning the mempool -- just evict-on-read if the transaction is now invalid (that's how most caches I've written work any way)
21:32 arubi yea but what about ones already in mempool that decayed?
21:32 reardencode regarding reduced fungibility: once confirmed these transactions have no different validity than any other - admitedly they are completely valueless when 0-conf
21:32 reardencode arubi: that's what I mean - when you read a tx from the mempool, you can check at that time if it's decayed past its validity and evict it
21:32 reardencode no need to scan proactively
21:33 arubi well if you're mining blocks you're pretty much updating mempool constantly
21:33 arubi I mena, you are always, but requesting block templates
21:33 reardencode sure, and I'm sure you check txs as you read them to see if a new transaction has (for example) spent one of that transactions inputs in the meantime
21:33 arubi only if a block comes in would you need to check that
21:33 reardencode so this adds one additional (and easy) check as you are reading your mempool any way
21:34 reardencode how do you reconcile the statement that a transaction never becomes invalid after it's been valid with RBF?
21:34 arubi both versions are valid
21:35 arubi until a block is mined. just not together of course
21:35 reardencode fair
21:35 arubi I'm sure you can code it, but it goes against transaction validity rules as they are in bitcoin right now. it's a pretty big change
21:36 reardencode yeah, I didn't realize that assumption was so deeply relied upon (although I should have, any current state of a system will become relied upon by someone)
21:37 reardencode ok, if you still have a minute, let's take this back a level: I think that there's a pretty big problem with bitcoin currently, because transactions can (and did in recent months) languish in the network mempool for many months. This makes using bitcoin difficult, especially for making exchanges that are not denominated in bitcoin (due to exchange rate changes).
21:38 reardencode I think that offering transaction authors a way to expire their transactions after a _defined_ period, instead of whenever the mempool owners want to is a powerful tool to enable new payments usecases
21:40 arubi it would make some coins worth less than others,
21:40 reardencode arubi: I still don't understand that claim, can you elaborate?
21:42 arubi coins that can expire are not only 0-conf, but potentially no-conf. that's pretty dangerous to accept as payment
21:42 reardencode we already have that distinction with non-RBF transactions and RBF transactions
21:43 reardencode it's just a sliding scale of reliability for 0-conf, it doesn't change the value of the coins once mined
21:43 arubi not really, any transaction can be replaced, but as long as it wasn't the first payment is always good
21:43 arubi there's not difference in reliability between rbf and non-rbf. consensus rules are the same re. replacement
21:44 arubi with decaying transactions, there's both risk of double spending before confirmed, and a risk of it just disappearing. there's no way to operate on such coin with confidence before it's confirmed
21:44 reardencode hmm, I may not fully understand RBF rules, can't I RBF a transactoin with one that pays me the entire input?
21:44 arubi you could do that both with rbf and non rbf
21:45 arubi rbf is just policy. the actual consensus rules are no different between rbf and non-rbf
21:46 reardencode I don't think that distinction matters here: If I make an RBF transaction that pays Alice 1BTC and then she makes a transaction paying .5BTC to Bob, but I RBF the original so that it only pays Alice .05BTC, how's that any different than me making an expiring payment to her?
21:46 reardencode in either case if the original transaction doesn't end up in a block, Alice is screwed
21:47 reardencode relying on 0-conf transactions is always a gamble, bigger with RBF an deven bigger with an expiring transaction
21:47 arubi rbf is irrelevant here. any 0-conf is vulnerable to double spending. a decaying transaction is vulnerable both to double spending and just disappearing
21:48 reardencode how about this: any transaction is vulnerable to just disappearing if its fee is low and most mempools are full
21:49 arubi not true. sender and recipient will continue to broadcast it
21:49 reardencode right, recipient can rebroadcast if they've saved the full TX
21:50 arubi their wallet does that already. at least core does
21:50 reardencode anyhow, I agree that expiring transactions reduce the strength of 0-conf, but 0-conf is already a crapshoot, so I'm still not seeing why deterministic time limits on the tx would reduce fungibility at all
21:51 reardencode if your payer is cooperative, they can resend, and if your payer is cooperative, they won't double spend - if your payer is malicious, you're screwed in either ase
21:54 arubi I'm trying to think if say a block was mined without our decaying tx in it and MTP is now past the decay date
21:55 arubi can a miner re-org that block and set time so that MTP on that block == the MTP that was on the last block
21:56 arubi now they can include the decaying tx since it hasn't decayed yet
21:57 reardencode good question - I'd need to review MTP rules
21:59 arubi with cltv\csv it's okay because the transaction becomes relay-able as mtp crosses the thing encoded in nsequence, so it's only the /next/ block that could include the timelocked transaction
21:59 arubi so no issue in reorg since any chain that reorgs will allow inclusion of the relayed timelocked tx
22:01 reardencode this seems like an issue of reliability of the expiration mechanism: at what point can I be certain that no future block reorganization could allow my (supposedly expired) transaction in
22:01 arubi suggestions I've seen were putting a 100 block maturity on it
22:01 reardencode so the MTP mechanism uses the median of the psat 6 blocks timestamp which is designed to resist reorganization attempts
22:01 reardencode er meidan of the past 11
22:01 arubi block maturity-like rule. like on generation txs
22:01 reardencode which puts it typically at the 6th past block
22:02 arubi right
22:02 reardencode I'll have to think harder about how that impacts my proposal for sure, because if there were a way for a malicious miner to include my tx _after_ I wanted it to expire, that'd be bad.
22:04 arubi basically any bad enough reorg can do that. all depends on how hard it might be
22:04 reardencode sure, it's all probabilities, just like relying on any confirmed tx ;)
22:05 arubi if enough people use this mechanism, it might be a good attack vector by a malicious miner against a lot of people at once
22:06 arubi either intentionally skewing the clock forward so txs decay faster, or reorg to do the other thing ^
22:08 reardencode yep, given that we have the blockchain, which is theoretically immutable after sufficient time, we should be able to define a reference point against which we can apply consensus rules for a mechanism like this
22:09 arubi that's where imposing maturity on these transactions comes in
22:09 reardencode in the worst case, using block height would be definitive, but given the broad skew of block durations, I'd like to offer MTP as well
22:10 arubi but payments that need to mature in order to become useful are less attractive to any recipient. what if they want to sell "now"?
22:10 arubi so that's where fungibility is hurt imo
22:10 reardencode right, so your concern now is once an expiring transaction is mined, can a malicious miner reorg the chain to make it impossible to ever mine that transaction
22:12 arubi right, without there being a double spend of it
22:13 gmaxwell 13:17:05 < reardencode> I'm working on a BIP to enable transaction inputs to expire if they aren't mined by an MTP/height, anyone interested in
22:13 gmaxwell any such proposal is probably dead on arrival, becuase they break monotonicity. and also probably an insult to the community because they suggest you haven't botered reading up on the 50 other times they've been proposed. :-/
22:14 gmaxwell every other time its come up the conclusion seems to be that they'd be okay if they required maturity on respend, but then no one wants the feature anymore.
22:15 arubi there's any written proposal? tbh I never saw one
22:15 arubi just sparse logs about it
22:15 gmaxwell Generally it seems like the primary motivation for these things is a misunderstanding of distributed systems-- the idea that there is a singular 'now' that things could expire by... but thats unphysical.
22:15 gmaxwell arubi: it's been on the list several times before as well as bitcoin talk.
22:15 reardencode gmaxwell: :( sorry, I'm relatively new to _bitcoin_ development and admit to not searching enough on prior work
22:17 reardencode gmaxwell: hah, and you've answered in the past
22:17 reardencode arubi: ^ for written similar proposal
22:17 gmaxwell it's not so much your fault.
22:17 arubi oh there we go :)
22:17 gmaxwell there is just a bit of death by 1001 cuts.
22:17 arubi "otherwise innocuous reorganization makes all its ancestors forever non-confirmable, even absent a double spend" oh verbatim
22:18 gmaxwell Perhaps there are new idea that could make some of these things more useful, but we never get to them because people show up and rehash the same discussions over and over again.
22:18 gmaxwell And any one person doing it is no big crime.
22:18 gmaxwell but after almost ten years of it, the cumulative weight is exhausting.
22:19 gmaxwell and it causes serious long term practitioners to stop reading new proposals, which is unfortunate.
22:19 reardencode gmaxwell: Yeah, I'm deeply interested in solving the problem of "My transaction is stuck for 3 months" for both payment processors trying to offer meaningful exchange locks, and users
22:20 reardencode I absolutely should have googled harder for ealier proposals, and will try to come up with something that doesn't require 100 block maturity
22:20 arubi maybe you could put an encoding of a signature and a txout that pays back to yourself in the witness :)
22:20 arubi something like gmaxwell's "would just be double-spent before they are considered fully expired"
22:21 gmaxwell reardencode: but I think thats why no progress is made; it /might/ be unrealistic to do anything like this without a maturity.
22:21 reardencode arubi: yeah, build a tx that can be spent by the receiver, or after "expiration" pays to the sender
22:21 gmaxwell or more generally: it might be impossible to do without a significant cost.
22:21 gmaxwell Free lunches are few and far between.
22:22 reardencode gmaxwell: yeah, I know I'm new here, but that part I'm aware of.
22:23 gmaxwell "if after_time x else y" is non-monotone. "y or if after_time x" is monotone. The latter is 'can be spent by y until after a time it can be spent by either x or y'.
22:24 arubi so 'expire or if after_time refund' is monotone?
22:26 arubi rather 'can be spent by y or if after_time can refund x or pay y'
22:26 arubi ah no :(
22:27 arubi why pay y if you want it to expire. you could say you just refund x, making it a double spend
22:30 gmaxwell what people want to accomplish is an expiration where no transaction fee is required.
22:31 gmaxwell I /think/ this is fundimentally not possible without breaking reorgs (the maturity requirement just gives you a requirement that at least any reorg that breaks it is just some implausably large number of blocks)
22:32 reardencode now that I understand the reorg implications, I agree that expiring transactions enforced by consensus have the same properties as mined coins.
22:44 reardencode although the more I think about it, for the payment processor scenario, waiting up to up to 17 hours to release funds (depending on the processors risk tolerance) doesn't sound unpalatable and in the usual case (say 12 block expiration, mined in 3) the realistic level of certainty provided is quite high
22:45 reardencode it does make limit the use cases for expiring transactions significantly from what I had original envisioned though.
22:45 reardencode (I / everyone else who's thought of it over the years)
22:57 reardencode active discussion on zcash
23:07 gmaxwell zcash is just kinda broken to begin with, their shielded transactions are busted and can't even reorg by one block
23:07 gmaxwell and they at least previously were going to deploy some consensus rule that nodes won't reorg by more than a couple blocks which opens up some pretty amusing attacks and random failures.
23:08 reardencode color me unsurprised -- bitcoin (appears to be) the only cryptocurrency developed by adults ;)
23:09 gmaxwell Zcash is weird because their crypto privacy stuff is very interesting... it's just all the other stuff is half assed.
23:09 gmaxwell and mostly just optimized for pumping.
23:09 reardencode yep -- I originally got super excited about zcash because of the privacy
23:09 gmaxwell e.g. their difficulty control code is totally unstable and even with constant hashrate swings wildly. ... they deployed it without even doing the most trivial simulations of it.
23:10 gmaxwell even bcash did a better job, and they're also pretty crazy. :)
23:10 reardencode wow, low blow at the zcash folks
23:11 gmaxwell hah
23:11 gmaxwell Well it's just kind of embarassing for everyone.
23:12 contrapumpkin well, even the snark stuff is pretty impractical for them isn't it?
23:12 reardencode I swear I'll give up eventually, but it seems like expiring transactions could still be pretty useful in scenarios like end of 2017, ie. it's better to try to send a tx that expires tomorrow, and is then in an unspendable state for another 17 hours, than to have your coin stuck in limbo for 3 months
23:12 gmaxwell reardencode: yes, no disagreement that it could be useful. But man, there is so much stuff that would be _awesome_ so long as you ignore the costs; or ignore that it has to work in an adversarial setting. :)
23:13 reardencode so in a common wallet use case, you might set an expiration of 200 blocks, and if it confirms within 100 blocks, no special rules are applied to spending the results, but if it takes > 100 blocks to confirm, then it must mature for 100 blocks before spending (like a coinbase tx)
23:13 gmaxwell contrapumpkin: they're going to hardfork to replace it with newer stuff which is less unrealistic, though it doesn't sound like they're going to force txn to use it still.
23:13 contrapumpkin ah
23:14 reardencode gmaxwell: yeah, I'm not trying to get a free lunch here, I'm seriously invested in both the political, economic, and technological future of bitcoin, and I think that this is a problem with solving in the protocol
23:15 gmaxwell reardencode: but what if it confirms at 100, and gets spent again at 100... then there is a reorg, and now the spend again is borked.
23:15 gmaxwell That seems bad to me.
23:17 reardencode hmm, right, which means you might as wall just always wait or it to mature in the first place
23:18 reardencode gmaxwell: any reference on why monotonicity is super important to the threat modeling of bitcoin?
23:19 reardencode I'm trying to ponder the implications of having transaction expiration and letting the market sort out how many blocks deep / how long before its expiration a tx needs to be mined to make those coins full fungible
23:19 gmaxwell not just threat model, it's necessary for the system to come to consensus. Reorgs of any depth are possible (but fall off in probablity that decreases exponentially with depth under some somewhat plausable security assumptions).
23:20 gmaxwell the only time the potential reorg depth becomes hard bounded is if you assume mining is centeralized.
23:21 reardencode ok, I think I'm starting to get it actually -- what you don't want is to have fork-A coin and fork-B coin that accidentally can't come back together because transactions that some people care about can only exist in one or the other
23:22 reardencode hence the coinbase must be deep before it can be spent so that miners don't have huge incentives to make persistent forks all the time
23:23 gmaxwell not just incentives, so that it doesn't happen just by chance.
23:23 reardencode mhm
23:24 gmaxwell e.g. if there was no maturity and mined coins instantly went into circulation, then there could be a two block reorg (which happen just by chance from time to time, and are guarenteed to happen if there are sizable internet partitions) -- and then an unbounded number of transactions by third parties could be invalidated. (well the number of confirmed txn by third parties that would be invalidated wou
23:24 gmaxwell ld be bounded by the block size, but still perhaps huge).
23:25 gmaxwell it would mean that even if you trusted your counterparties quite a bit you couldn't regard transactions as confirmed until they were pretty deep.
23:25 reardencode right, and similar would apply to expiring transactions, especially those mined close to their expiration, they would need to be pretty deep to be trustworthy
23:25 gmaxwell also, these sorts of events can destroy coins, since you might have discarded the private keys required to reissue new transactions.
23:26 gmaxwell Right, so one alternative idea that we've previously discussed is that you could have every txout carry with it the height of the highest non-monotone event in its history.. and any otherwise monotone txn that consume some set of inputs would propagate forward the maximum of the inputs heights as the heights on its outputs.
23:27 gmaxwell Then every output would be colored by it's non-monotone taint, and you could consider that when accepting payments. But even ignoring the complexity of doing this, there is a concern where it would create a race to the bottom, just like we've seen with confirmation counts.
23:28 gmaxwell E.g. many exchanges accept 3conf as irreversably confirmed even from more or less anonymous customers, even for very high value transactions... but this is kind of absurd, considering that 3deep reorgs have happened many times in bitcoin, and with mining centeralization at is is there are many different pools that if hacked/compromised could easily produce three deep reorgs.
23:28 reardencode right, and then you have people trusting the new 1-confirmation coins (ie. coins which had an expiring transaction in the last block) and then they get all pissed off when their value is destroyed
23:29 gmaxwell yea... so there would be commercial pressure to accept shallow coins, and people would because MOST of the time it's fine.
23:29 gmaxwell and then everything blows up when a once-in-a-year event happens.
23:29 gmaxwell so we're better off if the system just doesn't allow it.
23:30 gmaxwell (you could imagine that it would be plausable to design a cryptocurrency where you could also trade debt as equivilent to coins, and it would work fine until someone fails to repay the debt, and then you get a big system wide failure)
23:30 reardencode so the difference between this and a run of the mill double spend is that the run of the mill double spend attempt would require the double spender to collude with the miner to insert the malicious transaction into the bottom of the reorg?
23:30 gmaxwell (so for everyone's benefit bitcoin requires transactions be presently solvent. :) )
23:31 gmaxwell reardencode: right a double spend requires malace by both the double spender and a reorg.. and only impacts decendents of the double spend. Whereas a reorg break doesn't even require malice at all by anyone (though malice can make it more likely), and it breaks every non-monotone thing, not just single transactions.
23:36 reardencode gotcha, thanks for taking the time, I really appreciate it
23:37 reardencode gmaxwell: so, I'm a reasonably clever software engineer who would like to contribute to bitcoin, what's a better direction for me to invest my effort other than trying to make transactions expire?
23:38 gmaxwell Well working on consensus protocol changes is the worst way to get started, you'll go splat fast.
23:39 gmaxwell it's better to first become really well informed about the system by working on other stuff.
23:39 gmaxwell Then at least your ideas on protocol changes can be constucted and presented in a way that will be better accepted.
23:40 reardencode gmaxwell: btw, I actually had written my thoughts on this up in full bip format:
23:40 reardencode (not that I expect you to read it, but just that I did get pretty deep into understanding the protocol and prior BIPs before bringing it up in here)
23:41 gmaxwell there are now so many 'bips' being written without being well informed or discussed I worry that BIPs in general are going to get abandoned.
23:42 reardencode gmaxwell: doesn't surprise me -- I was trying to figure out how best to start and someone on reddit suggested writing it up in BIP style as a starting point, guess I was misled by a redditor ;)
23:44 gmaxwell hm interesting.
23:45 gmaxwell I haven't thought about why it was happening. There are some factors that I was aware of.. e.g. the general idea about bips is that anything could get published, but since discussion is required first people would not bother going forward with non-viable stuff once they saw from discussion that it wasn't viable.
23:46 gmaxwell That worked for a long time, but a couple things changed, one is that because of political drama the bar for discussion became ultra low. e.g. stuff where no one bothered responding is getting assigned numbers now.
23:46 gmaxwell Also, because of people getting worn out it's more common that rehashed or broken proposals just get ignored.
23:48 midnightmagic Maybe a two-tier. The first drafts aren't BIPs but instead RFC or something.
23:48 midnightmagic Moved up to BIP once sufficient comment is acquired.
23:48 gmaxwell oh also this channel is 99.9% dead.
23:48 gmaxwell which probably doesn't help.
23:48 gmaxwell this is the first discussion of over a couple lines here in a long time.
23:48 midnightmagic It doesn't have to be anymore; the Prior Problem has been solved.
23:49 gmaxwell yes but the stink remains.
23:49 midnightmagic But .. right. I was going to say.
23:49 gmaxwell Actually I only bothered rejoining here to monitor for a while to decide if I should forcably shut down the channel or not.
23:50 gmaxwell so far been leaning towards shutting it down... almost not traffic, and what is here is mostly people shouting into the void. :)
23:50 midnightmagic I'm sure nanotube would be amenable to pretty much whatever you guys want to do with it.
23:50 gmaxwell Sure.
23:50 midnightmagic (Which I guess didn't need to be said but typing is a habit.)
23:51 gmaxwell Based on the past I'm uneasy about actually using it again, I mean having technical control and historical mandate for -wizards hasn't prevent it from being a drama vector.
23:52 midnightmagic That was mostly in the name. Seems to annoy the pathological ODD types.
23:52 gmaxwell I don't think so.
23:52 midnightmagic (imo)
23:52 midnightmagic No?
23:53 gmaxwell I think the non-viability of large unmoderated public discussion channels for useful discussion combined with a believe of entitlement to apparently public resources is the issue.
23:53 midnightmagic Entitlement -- yes. "I'm a wizard, Harry."
23:54 midnightmagic ODD and massive entitlement.
23:54 gmaxwell midnightmagic: right "I'm a developer, Harry."
23:54 midnightmagic heh
23:54 gmaxwell we fix that with -core-dev because the set of people who might feel entitled is acceptably small.
23:55 midnightmagic Channel name matches the topic. I'm pretty surprised how effective that is, personally. There seems to be some respect that engenders straight off, with a small corner.
23:55 gmaxwell I think a better fix which I haven't tried yet is to replace channels with ones with clear owners; to eliminate the entitlement problem.
23:55 contrapumpkin #gmaxwell
23:56 gmaxwell Pretty much exactly that. something like #gmaxwells-den and then no one gets bent when gmaxwell sets the freeking rules.
23:56 midnightmagic I believe that would make people think they were ego-busting.
23:56 midnightmagic (To invade it.)
23:56 gmaxwell well you just ban them. I never minded banning people in wizards or here, the annoying drama is the whitenighting after the fact.
23:57 gmaxwell By people who are usually okay but aren't yet worn out by the borderline behavior and haven't really internalized that if the main anchors of the venue walk the venue is dead.
23:57 midnightmagic I still think a multi-level set of introducers would mean that, e.g. I could randomly invite a bunch of people who've asked me, and then they invite people who ask them, and so on until the shitheads appear. Invading that set would be way more time-consuming than just registering a freenode account.
23:58 gmaxwell Having open venues has a lot of advantages. Including the fact that the people with the most to offer are not going to jump through a lot of hoops.
23:58 midnightmagic Yes. Anchors. That was something I've grasped pretty much right from the start. I've often, often, repetitively said that if e.g. sipa goes away the point of the channel-under-discussion pretty much goes away.
23:59 gmaxwell I /think/ making ownership explicit is virtuious because it's almost always implicit anyways-- e.g. this channel died virtually completely when we left. 10000 fold reduction in daily traffic, easily. Effectively the anchors owned it, but got abused when they wanted to do whatever was needed to keep it usable for them.