Transcript for #bitcoin-dev 2017/12/27

00:03 eck very insightful, thanks for letting us know
00:09 spectra |shame on the devs for selling out
00:09 spectra |why don't we just increase the block size?
00:17 eck this topic is more appropriate for #bitcoin
00:29 xexe can any set of rules at all governing the txes cherry-picking from the mempool be introduced as protocol for miners?
00:30 Dagronmaster "cherry-picking" transactions is a feature, not a bug
00:31 xexe who defines the cherry-picjing rules at present?
00:31 eck the person who mines the block
00:31 xexe that's crazy, shouldn't they have just the validating funciton and not governing?
00:33 Dagronmaster who's governing?
00:33 xexe what if we figure out some probability distribution weight function let's say add a bit of randomization here. how can this be enforced for miners then?
00:34 eck who would enforce it?
00:34 Dagronmaster from each miner according to his ability, to each transaction creator according to his need
00:35 eck transactions in the mempool are not globally consistent anwyay, so it's pointless to try to enforce
00:35 xexe the community, peeps who actually use the network, devs etc. it's about the governance model isn't it
00:36 eck so what would you propose they do?
00:36 eck we can have a twitter poll to ask people what transactions should be included in the next block but i don't think that's going to help
00:37 xexe well they whould be fairly consistent, or tend to be at least, nevermind the propagation latency and all that
00:37 eck in the common case, yes
00:38 eck there are legitimate reasons a miner might want to do something out of the ordinary though
00:39 eck bitcoin is designed to give miners the economic incentives to do the Right Thing, which is really the best you can do in a distributed, trustless system
00:39 eck it might not work 100% of the time but it's pretty close
00:39 xexe the q is how this or any other model be introduced tyo miners. what sowtware code do they use? core or their own?
00:40 eck most of them run a bitcoin core node on the edge of their network, but in some cases they might modify the code
00:41 eck here's an example of a legitimate reason you might want to do something unusual as a miner. let's say you're mining, and for some reason you have a lot of "dust" utxos (e.g. because you are also an exchange). you might want to mine a block to coalesce your dust utxo outputs with 0 tx fees, and you probably wouldn't even bother broadcasting those txs to the network until you actually mine a block with
00:41 eck them.
00:42 xexe i think they have more than enough economic incentinve at the moment with 4+ in each block
00:42 eck the example i just gave is not even hypothetical, it does/has happened on mainnet
00:44 xexe it's just that economic incentive alone has never been the best governing model to do the Right Thing in this life afaik
00:45 eck maybe not, but that's how distributed systems work
00:46 xexe we are talking about chaos systems, where probalbility laws rule and distribution fucntions and strange attractors and all that
00:46 eck how are strange attractors and chaos systems at all related?
00:46 xexe and at the oment i do not see any quantum probabalistic features employed in this cherrypicking
00:47 eck you're right, you can implement it with a single for lop
00:47 eck consider this though, let's say there are some bad actors in the system. that should be ok, as long as they don't have excessive hashing power.
00:47 eck which is why mining decntralization is important
00:48 eck and decentralization of the network in general
00:49 xexe you mean ddosing the meemppol with dust? is this shear fee amount is really the only option to mitigate that sort of attack out there?
00:50 eck for the people creating the dust transactions, or the people mining the blocks
00:50 eck ?
00:58 xexe miners have had too much power here, and that should better be limited for them. they should basically run on 'donations' maybe not quite but something like that
01:11 xexe the same with these powers that be, no matter how much it's never enough for them. they simply do not have such self-limiting function in their psyche. they are there in society because they have some mean function but that's all. they have never been the most wise or pure (and they act as if they are) in fact they are the most vile & vicious minority of the whole hive. they insatiable hunger for power &
01:11 xexe control is going to ruin evrything. that's why the hive should figure out some way to prevent them from seizing this absolute power time & again in history..
01:36 echeveria xexe: you simply can't attempt to force rules over transaction selection.
02:14 xexe yes enforcement isn't the dharmic way or method to achieve anything worthwhile, distributed consensus is..
02:15 echeveria yes, in bitcoin that happens after transactions are confirmed.
02:15 xexe i'm telling you why i'm trying to do. not how i'm trying to do..
02:16 echeveria I'm telling you, no matter how noble, you can't have this.
02:23 xexe i'll think about that but if the hivemind isn't going to figure out that then perhaps we will forever be at the mercy of miners, which isn't much sustainable state of affairs in the long run..
02:24 vinnix I was looking into "bitcoin/src/leveldb/db/" function DBImpl::Get
02:25 vinnix there is a interesting part "if (have_stat_update && current->UpdateStats(stats)) { MaybeScheduleCompaction() }
02:27 vinnix since I got the mem leak report from valgrind, I am diging more into the code :)
02:28 vinnix from the user perspective I'm observing it during '-reindex'
02:43 echeveria xexe: you can wax lyrical about this all you want, but the simple reality is that miners will choose whatever is profitable for them. trying to make social rules around transaction selection is worse than useless.
02:49 xexe can you still afford to use the fintech in question? because i can't anymore; which isn't much egalitarian status quo..
02:51 Dagronmaster to look for egalitarian qualities in anything remotely libertarian is a recipe for disappointment
02:56 xexe yes heart-breaking it is, as ususal. but wax candles & noble fir is so much holiday..
03:29 xexe not even mentioning lack of deflationary liquidity this whole enterprise resembles more and more some enterprise for Argonauts instead of sustainable ecosystem; but perhaps it was meant to be has always been so from the start so at the end of the day it's al-right..
04:32 xiedeacc what's nChainWork for ?
04:33 xiedeacc in class CBlockIndex
05:03 phantomcircuit xiedeacc, it's the inverse of difficulty
05:03 phantomcircuit so you can easily calculate the total work done in a long chain
05:16 DSidH I realize that we can get two public keys from each signature. How did arubi say we can get 4 or even 8?
05:16 DSidH And I am not able to find any documentation for encoding signatures with recovery.. except maybe this one:
05:17 DSidH can anyone guide me on how to encode a recoverable signature?
05:18 xiedeacc ok, thanks~
05:37 echeveria DSidH: read the comments in the libsecp256k1 code.
05:53 DSidH echeveria: I did Google for "libsecp256k1 code". Too much info. Can you please point me to the right github link?
05:55 DSidH one link is this:
05:57 DSidH but it does not give too much info about encoding
05:58 DSidH [10:46] <DSidH> I realize that we can get two public keys from each signature. How did arubi say we can get 4 or even 8?
05:58 DSidH actually its 4 (2 x values and 2 y values for each x).. but I don't get how it can be 9
05:59 DSidH 8*
07:03 echeveria DSidH: it’s in the “bitcoin” group on github.
08:14 ujjwalt Hi guys
08:14 Randolf Hello ujjwalt.
08:14 ujjwalt Anyone here can help me understand how base58 in bitcoin works at the implmentation level
08:14 ujjwalt Hi Randolf
08:15 Randolf Base58 is merely a representation of a number.
08:15 ujjwalt I specially can’t seem to understand what this line does - `std::vector<unsigned char>::iterator it = b58.begin() + (size - length);`
08:15 Randolf Just like Base16 is (hexadecimal).
08:15 ujjwalt Right. How does the bitcoin version work?
08:16 ujjwalt Not strong at CPP so can’t seem to understand some things
08:16 Randolf Base58 is a standard. There's no "Bitcoin version" that differs from any other use of Base58.
08:16 ujjwalt Like this: `int size = (pend - pbegin) * 138 / 100 + 1; // log(256) / log(58), rounded up.`
08:17 ujjwalt first we’re counting the leading zeroes in the data buffer
08:17 ujjwalt when encoding
08:17 Randolf A zero is encoded as a 1.
08:18 ujjwalt ok
08:18 Randolf ujjwalt: Maybe this document will help:
08:18 ujjwalt I’ve gone through it
08:19 ujjwalt just clarify two three things for me in the code
08:19 ujjwalt / Apply "b58 = b58 * 256 + ch".
08:19 ujjwalt what are we basically doing here
08:19 ujjwalt and why are we going over b58 in reverse?
08:19 ujjwalt and this line: std::vector<unsigned char>::iterator it = b58.begin() + (size - length);
08:19 ujjwalt Sorry how am I supposed to paste code here?
08:20 ujjwalt Is there a convention we follow on this room?
08:20 arubi it's fine to paste just a couple lines
08:20 Randolf If it's only a few lines, then IRC chat is fine,but for longer code use a service like
08:20 ujjwalt got it
08:23 ujjwalt why do we use a revese iterator? Any idea?
08:28 arubi DSidH, it can be that 4 pubkeys can be recovered from a signature and message if the 'r' value in the sig is smaller than (p - n) (p is the prime and n is the curve order), because we take the original R's point X coordinate (mod n), so when we have such an r value, we can check if it's a valid X coordinate, and also check if (r + n) is a valid coordinate, and recover 2 different pubkeys for each option
08:28 arubi ujjwalt, I'm not too sure about the c++, but wouldn't you do any base conversion like that?
08:30 DSidH arubi: so we have r, r+n and the corresponding y's?
08:31 DSidH (I mean the key recovered from r and r+n)
08:31 arubi the Y coordinate inclusions in the pubkey hash or not is what makes for two addresses for each recovered pubkey
08:32 DSidH ok.. I am referring this section 4.3 to implement key recovery
08:33 DSidH where the cofactor is 1 as in bitcoin
08:34 arubi so I didn't understand the question. the pubkey being compressed or not matters for bitcoin specifically. that document doesn't care about it
08:37 DSidH nvm the question, I was thinking that we find two different x coordinates first for possible public keys. Then each x has two possible y coordinates, which gives us 4 points
08:38 DSidH similar to we find the y of a sig.. but this seems to be wrong
08:39 arubi yea the Y values of the recovered keys are unrelated between them
08:39 arubi or maybe better, are not +-Y :)
09:26 DSidH arubi: I am finding that most signatures have only 2 keys.. is this expected?
09:27 arubi here:
09:27 arubi first one should recover 4 keys to 8 addresses, second one recovers only the "small" R_x pubkeys, third one should recover only the "big" R_x keys
09:28 DSidH R = 68053886848776677441133659300543143116349777749036069675459047078098205334997 S = 40438720191521726420796838536941300170400936095250279510361216891756193879060 Hash as Int = 20329878786436204988385760252021328656300425018755239228739303522659023427620 PK1 105932108814481395905759851533798966534115425465770952541697246848027772578198,80011604461019602521402058374560558590524450437601070093989309960612944029061 PK2 10730358229073
09:28 DSidH example
09:29 arubi err, s/first/third/ and third/first. seems like the gist is backwards
09:29 DSidH 1 min checking
09:53 DSidH arubi: ah I get it now. we get max 4 keys in terms of points and then 4 more considering uncompressed and compressed!
09:55 arubi yep, that's it
10:48 DSidH arubi: thanks for the test vectors. I am still confused as to how to decide which public key to associate the signature with.
10:48 DSidH all of them return true
10:58 arubi yes, the first byte in the 65 byte recoverable signature tells you which to choose. that's the 3x8 table I posted yesterday. if you base64 decode the signatures, you'll see it
11:12 DSidH ok so there is some canonical ordering.. first do compressed using r then r + n and then get 4 points, then do same for uncompressed
11:19 arubi also the even R's recovered key is returned first
11:38 DSidH arubi: this one :)
11:39 DSidH so I have to strip the first byte of the signature
11:41 arubi yes those signatures are [ recid byte ][ 32 bytes r ][ 32 bytes s]
11:46 DSidH arubi: is the even-odd guaranteed? I could not see it directly, like can we not have (y_even, x < r), (y_even, x < r)
11:47 DSidH (r = n)
11:47 arubi I'm talking about R's y being even or not
11:48 DSidH oh ...
11:48 arubi you take 'r', check if it's a valid x, find it's two y values and recover right, so the pubkey that gets recovered by the r_x,r_y where y is.. yea you get it :)
11:49 echeveria more ideally we should just have a consensus rule that says y must always be even, then the pubkey becomes 16 bytes.
11:49 DSidH yup I was confusing with the xs and ys of the pub key
11:49 echeveria too late for that though.
11:50 arubi you mean 32 bytes?
11:50 echeveria er, yeah.
12:01 DSidH or even better public key should just be x coordinate
12:02 DSidH and yes even
12:02 DSidH but can we not ignore the y when hashing?
12:03 DSidH so then question of "even" will not even arise
12:06 arubi in bitcoin the public keys are compressed
12:07 arubi so it's 33 bytes. only very old wallets and services might use an uncompressed pubkey these days
12:07 DSidH yes but we just assume public keys are only x coordinates and when we need to verify, we always take the even y
12:07 DSidH but while computing Hash(pubKey) we only use x
12:08 arubi but what if your pubkey actually has an odd y coordinate?
12:08 DSidH its a wrong key
12:08 arubi you mean when you generate the private key itself, you also tweak it to always make an even point?
12:08 arubi even y that is
12:08 DSidH yup we always assume that odd y is invalid
12:09 arubi well it at least cuts the private key range by half :)
12:09 DSidH if Y is odd then -Y will be even
12:09 arubi sure, but then you're losing security bits I think? not sure
12:09 DSidH hmm no the private key space is still same
12:10 arubi how so? you also have to negate the private key
12:10 arubi if Y odd is invalid, and x makes that point, then it's an invalid private key
12:10 arubi but -x will be the valid one
12:10 DSidH hmm maybe Im confused. We generate x and then compute Y = xG
12:10 arubi and because you're always reducing mod n, then half the range is always flipped
12:10 arubi no no
12:10 DSidH and if Y is even (i.e., Y_y is even)
12:10 arubi xG = (X,Y)
12:10 DSidH then we set Y = -Y
12:11 arubi better, normally the private key is 'd'
12:11 DSidH no capitals are curve points in my notation :)
12:11 arubi so dG = (x,y)
12:11 arubi oh, yes, so xG = Y and -xG = -Y
12:11 DSidH dG = (x, y) and -dG = (x, -y)
12:11 arubi yes
12:12 arubi so if dG = even y value, then -dG = odd y value
12:12 arubi makes -d % n an invalid key, which is half the range
12:13 DSidH so we keep private key d and keep pub key either -dG or dG
12:13 DSidH depending on which is even
12:14 arubi how would that work? someone with the key d and someone else with the key -d will have the same pubkey?
12:14 DSidH as far as I see this does not reduce private key space, since every d will generate a valid pub key
12:14 DSidH then that one with -d and d will have lot more to worry about then pub key collision :)
12:15 DSidH two people generating the same d wont happen i
12:15 arubi their first worry would be this new consensus rule
12:15 arubi since it's more strict than the same d. it adds -d too
12:16 arubi I don't see how that's not just throwing away one bit
12:16 arubi if we say all pubkeys are 32 bytes + 1 bit, then you say lose that one bit
12:17 DSidH yes but the private key space is still 32 bytes so security is only 32 bytes
12:18 arubi it's essentially the same as half the range, if you make people sign with their -d when they have d
12:18 arubi since a signature from d will not be valid if the point dG is odd
12:19 DSidH it will be valid under pub key -(dG)
12:19 arubi it won't if you'll use your normal d to sign
12:20 arubi you'll have to use -d if d gives you an odd point
12:20 DSidH hmm maybe not in current scenario but if we only keep hash(x coordinate of pub key), it will be
12:20 arubi it still won't be
12:21 DSidH sorry I made a typo
12:21 DSidH signer should sign it using -d
12:21 arubi then they're effectively choosing from half the range..
12:21 arubi it doesn't matter that they generated d if they end up signing with -d
12:23 Eric_Fly anybody is here? I need help
12:24 DSidH hmmm in the malleability attack, didn;t the signature verify under both dG and -(dG) ?
12:24 arubi it's s and -s
12:24 arubi it's R's odd\even that gets flipped in that case
12:24 Eric_Fly what's the segwit address?
12:25 DSidH ok... I see but I still don't see how removing one byte can reduce security by half
12:25 DSidH something is wrong in the reasoning
12:25 echeveria DSidH: remember that's not the only cause of malleability.
12:25 DSidH instead of 256 bits we may get 255 bits
12:26 DSidH half of them being lost as u said
12:26 arubi really the security for secp256k1 is 128 bits
12:26 echeveria yup.
12:27 DSidH If I am able to guess -d then I am able to guess private keys of all two pub keys.. so security should not be lost
12:28 Eric_Fly I saw segwit address on blockchain like bc1..... but i decoderawtransaction like multisigaddress why?
12:28 echeveria Eric_Fly: that's bip173.
12:28 echeveria bech32.
12:28 arubi it's not about guessing. the attacks are much more involved than some brute force search
12:28 Eric_Fly I decode to bip173?
12:29 arubi I don't know for certain, I'm just worried making half the people negate their private keys for signing will make the attacks easier
12:29 DSidH in fact if I recall (with a bit of hunch) that security is only 128 bits because we only need to search half keyspace
12:29 DSidH in the generic group model
12:29 arubi I don't think that's the reason
12:30 DSidH sorry its square root of the keyspace
12:32 arubi I think it's related to the size of the largest factor in (n-1)
12:32 arubi I'm really not sure
12:33 DSidH in the generic group model, it takes approx sqrt(n) tries to find dlog.. you're right that its not related to the issue we are discussing
12:33 arubi maybe not
12:34 Eric_Fly thank u for echeveria~i try
12:36 DSidH (where n is our prime order of G)
12:38 arubi well, whether it's reducing security or not, it's still a lot of overhead just to save what's really one byte :)
12:41 DSidH ecc is always confusing. I'll go back to key recovery code and paste some test vectors
12:42 arubi cool
12:52 Eric_Fly how to deocde scriptPubKey's hex to Bech32 address
12:57 DSidH arubi: if r or s are less than 32 bytes do we pad with 0s?
12:57 DSidH in the key recovery encoding
12:58 arubi right
12:58 arubi left pad
12:58 DSidH kk thx
13:33 xiedeacc what's CBlockIndex* pskip; for?
14:58 Dagronmaster xiedeacc: It's a pointer to a predecessor block of this block
14:58 Dagronmaster a predecessor that's farther up the chain than the last one
15:00 xiedeacc thanks
15:00 xiedeacc after google
15:01 xiedeacc it like a binary tree, and predecessor is father(grand... node
15:01 xiedeacc just for speedup lookup
15:28 IR Hi, when creating a transaction, do we use a different scriptSig for each input?
19:06 DSidH arubi: need some help
19:06 DSidH
19:06 DSidH everything done .. recovery works.. except that my signatures don't match with test vectors. Both sigs are valid (at least with my code)
19:06 DSidH can you let me know which one are generated by you
19:07 DSidH test vectors are from a SO post
19:08 arubi alright let's see
19:08 DSidH I used dsha256(magic_bytes+message) as the hash
19:08 DSidH to be precise: [magicBytes size]++[magicBytes size]++[msg size]++[msg]
19:09 DSidH [magicBytes size]++[magicBytes]++[msg size]++[msg]
19:11 DSidH perhaps the test ones were generated before core used ref6979
19:13 arubi getting the same sigs as the ones in MySig
19:14 DSidH arubi: thanks mySig are ones I generated
19:14 DSidH any idea why the core test ones are different (at least the op seems to imply its from core)
19:15 arubi not sure, I'm assuming that if we tell core to sign those messages with the keys, it'll get the same sigs you and I are getting. how old are the sigs from the test?
19:15 arubi I mean, how old is the post
19:16 DSidH post is Mar 1 2014
19:17 arubi actually I'm not sure when libsecp was merged into core. I think it was before that.. can you link the post?
19:21 DSidH
19:24 arubi it reads that they generated these themselves and checked for formatting against core? not sure. anyway, for the first sig, the k was either 2AB46A84378337CC07F3A782BDD11E8D10285CA22521CC618FED49E900692133 or D54B957BC87CC833F80C587D422EE171AA8680448A26D3DA2FE514A3CFCD200E . I tried rfc6979 with the private key and double hash of the message, but that's not it
19:25 DSidH ok.. thx for validating this
19:25 arubi np
19:34 arubi DSidH, confirmed core is getting the same sigs as us
19:50 DSidH arubi: thanks. btw you were right, segwit is much easier to implement
19:51 arubi \o/ dat data re-use
19:51 arubi and witness stacks instead of scripts, oh man