Transcript for #bitcoin-dev 2017/08/31

01:55 nreefode hi all. I am writing an RPC to get up to speed with bitcoin-core. At some point I would like to be generating an empty (but valid) new block. CreateNewBlock pulls things out of the mempool, yes?
01:57 nreefode I'm thinking I can add a flag to CreateNewBlock so that it does not call addPackageTxs, which it looks like is what's grabbing txs from the mempool
02:01 nreefode oh you know what, the docs for addPackageTxs explicitly say "we don't remove transactions from the mempool as we select them for block inclusion"
04:17 esotericnonsense hm.
04:18 esotericnonsense my 'random source' has given me a bip39 seed with three subsequent words identical.
04:18 esotericnonsense :>
15:02 esotericnonsense after multiple hours of trying to figure out why my bip32 derivation code is not working properly against the test vectors i decide to use a different base58 encoder
15:02 esotericnonsense it was always fine
16:24 dviola hi
16:25 dviola I see my initial sync is going a bit slow today, I'm getting connection refused and such
16:25 dviola 2017-08-31 16:23:35 Peer=5 is stalling block download, disconnecting
16:25 dviola 2017-08-31 16:23:35 connect() to [2001:0:5ef5:79fb:14be:388:93d8:4eaf]:8333 failed: Network is unreachable (101)
16:25 dviola in the debug.log
16:25 dviola is it possible my ISP is messing with me?
16:30 dviola how do I know if my node is able to connect to other nodes?
16:33 dviola I wonder if that time out is simply because my node can't be reached from outside
17:05 dviola looks like that was a connection to ipv6 and I can't use ipv6
18:29 asimplecoder hey
18:29 asimplecoder anyone online?
18:30 asimplecoder i need some help with bitcoind
18:32 adiabat asimplecoder: people are online, just ask, preambles etc are generally ignored here
18:34 asimplecoder i cant be able to figure out how to use bitcoind's sendrawtransaction
18:34 asimplecoder bitcoin-cli sendrawtransaction mysignedtransactioninhex allowhighfees=false error: Error parsing JSON:allowhighfees=false
18:35 adiabat I think if you want allowhighfees, just put a true at the end?
18:36 esotericnonsense looking at bip32 - am I correct in understanding that a hardened public child key cannot be obtained from any public parent key or set of
18:37 esotericnonsense that is to say that it can only be derived by knowing some private parent key, determining the hardened private child key and then obtaining the point
18:37 adiabat asimplecoder: as in, don't say allowhighfees=false, just put false after the hex string
18:37 asimplecoder if i put true at the end i get:
18:37 asimplecoder error code: -25 error message: Missing inputs
18:38 asimplecoder same as with false
18:38 adiabat asimplecoder: same? that's a different error
18:38 adiabat try decoderawtransaction to see what's wrong with the tx
18:38 esotericnonsense e.g. m/0'/0/0' pub _cannot_ be obtained from m/0'/0 pub nor m/0' pub nor m pub
18:38 esotericnonsense you need any of those as prv
18:39 esotericnonsense (this logically follows from bip32 CKD's, just confirming that I haven't missed something)
18:40 asimplecoder its nothing wrong with the transaction
18:42 asimplecoder http://dpaste.com/2T4S9N4.txt
18:42 asimplecoder check it out^
18:44 adiabat the missing inputs error means the inputs your trying to spend aren't found
18:45 adiabat so the tx in isolation may be well formed, but the txo it's spending is gone
18:45 asimplecoder so it might have been gone throw?
18:46 adiabat the vin:txid:0 is probably not a utxo, you can query that with getrawtransaction
18:46 adiabat (if you have txindex=1)
18:46 arubi esotericnonsense, right, leafs on the same branch aren't related at all in any way
18:47 asimplecoder no it got the bitcoin on it
18:47 arubi and specifically hardened paths can not be derived from non hardened paths
18:47 adiabat asimplecoder: as a guess, bitcoin txids are "backwards", in that if you're writing your own stuff, the bytes have to be reversed
18:48 asimplecoder yupp its a custom script
18:48 arubi you can only go forward if you know the private key at all, and can only go backwards with a private key and a non hardened xpub
18:49 adiabat the txid in the hex string going into decoderawtransaction is backwards from the hex string you'll see from the json output
18:49 esotericnonsense arubi: well you can go forward from pub to pub with pub, just not to a hardened child
18:49 asimplecoder i tried to push and i got the error "Validation Error: Insufficient fee. Minimum fee is 1 sat/B."
18:49 arubi yea, that's right. on non hardened paths
18:50 arubi sorry, can only go forwards to a hardened path*
18:50 adiabat asimplecoder: reduce your output value a bit to up the fee
18:50 esotericnonsense i'm unsure what you mean by on non hardened paths. it seems that i could go from pub m/0'/0 to pub m/0'/0/0 just fine.
18:50 esotericnonsense yes, indeed
18:50 esotericnonsense can only go forwards to a _non_hardened path :P
18:50 arubi right, or a hardened path with knowing the xpub
18:50 arubi errr
18:50 arubi expriv
18:51 asimplecoder whaaat "Parse: exception decoding Hex string: String index out of range: 381"
18:51 arubi esotericnonsense, sorry, I'm only paying 20% attention, I'm confusing more than helping :) I think you got it
18:52 esotericnonsense lol
18:52 arubi try to look into the private key + parent xpub exploit, it's pretty cool
18:53 arubi as when a private key on a non hardened path is revealed, then you can derive parent xrpiv's until the non hardened path ends. I think it's a cool way of applying bip32 stuff :)
18:56 esotericnonsense yeah, i've looked at that
18:58 esotericnonsense it seems interesting in that you can create a wallet that uses only hardened children, generating say the first 100 pub/priv when the parent is available, then removing the parent (or locking it behind a more secure encryption passphrase than the children)
18:58 esotericnonsense you would only need to unlock the parent to generate more pub/priv
19:00 esotericnonsense i suppose that actually you could just keep the parent encrypted (both pub and prv) and use non-hardened
19:01 esotericnonsense basically it's like having a non-HD wallet with a keypool, but the keypool can be refreshed deterministically, then again it would be easier to just use different parent paths for this sort of thing
19:02 asimplecoder fucking hell error code: -25 error message: Missing inputs
19:09 arubi esotericnonsense, that's how core does it
19:10 arubi it's hardened hd only, and you have to unlock the wallet when the keypool is depleted to generate more
19:11 arubi really you can just generate a huge amount the first time and keep encrypted forever, but it gets pretty bit quickly with private keys wrapped in secp256k1 asn.1 for each key :P
19:12 arubi well maybe not quickly, iirc it's hundreds of megs at a million keys, which is actually 2 million because of the internal change
19:14 arubi electrum uses non hardened derivation, different from core. really most wallets use bip44 or now starting to use bip49 which are both public derivation
19:17 esotericnonsense yes, i'm looking in to bip44/49 now. wanted to cover bip32 first to understand how it works. have reimplemented bip32 in python, passing the vectors
19:18 arubi awesome. the newest added vector is due to a really nasty bugs some implementations had with a leading zero byte for private keys in the extended encoded, be sure to pass that ;)
19:21 esotericnonsense yep, all checked out first time :)
19:21 esotericnonsense (well, once I got any of them working)
19:26 arubi what are you using for ecc esotericnonsense ?
19:26 esotericnonsense python-ecdsa, which seems quite slow. but this is kind of a toy for now.
19:27 esotericnonsense (it is also slow because i don't optimize, I forget all previous results each time).
19:29 arubi cool, slow or not, if you're using it for exploring stuff, then the interesting bits are mostly interactive anyway :)
19:29 esotericnonsense i did write the bits of ecdsa I needed for doing this years back but i've lost the code and it was horrendous anyway
19:30 arubi heh yea, I wrote my own too. still using it. it's a lot slower than anything in python ever will be
19:30 arubi mine is in gnu bc
19:30 esotericnonsense lol
19:30 esotericnonsense ???
19:30 esotericnonsense :D
19:31 esotericnonsense i could never get on with bc. i tried. always just use python interpreter whenever i need to do quick math stuff.
19:31 arubi https://github.com/fivepiece/btc-bash-ng and bash to tie stuff together :)
19:31 esotericnonsense awesome
19:32 arubi cheers, if you need some exotic math stuff, I might have it. if I don't, let me know, I love to implement cool stuff
19:33 arubi it's not always 100% correct, but I try :P
19:33 esotericnonsense i'm mostly doing this to aid my understanding of how wallets work and what a 'good' wallet storage format should look like
19:33 arubi I guess I do too
19:33 esotericnonsense after having a fun time trying to extract funds from a bitcoinj wallet, compiling tons of java shit just to get the keys out
19:33 arubi wait I have something you'd like
19:34 arubi https://gist.github.com/fivepiece/74b6f8bfa8816ee7e283d25a36947629 , core's wallet with keypool=0
19:35 arubi I kinda want to dissect the wallet further, my real wish is it being able to use a 64 bytes value as the bip32 seed
19:35 arubi as the input to the bip32 master xpriv rather
19:36 arubi currently it's using one valid secp256k1 private key as the 32 bytes to the [hmac-sha512 "value" "Bitcoin seed"] thing
19:36 arubi if it could be a 64 bytes value, then it could also be an output off a bip39 kdf, and I'd be able to get my core's private key on a hardware wallet! :D
19:40 arubi brb, sorry, I try to drop this info onto anyone who has an interest in improving the wallet scheme. I think the current "hey just change the constant" method is difficult. already folks are asking wallet providers to be able to sign with a "segwit address"
19:41 arubi what I mean is, this needs fixing. it's getting to the point it's very hard to maintain..
21:25 esotericnonsense lessons in cryptography being full of footguns #229384583
21:26 esotericnonsense https://github.com/petertodd/python-bitcoinlib/blob/master/bitcoin/core/key.py WARNING: This module does not mlock() secrets; your private keys may end up on disk in swap! Use with caution!
21:26 esotericnonsense such an obvious possibility that I'd never even considered before
21:29 esotericnonsense i think at least core and electrum now offer the ability to encrypt on boot (otherwise you end up with an unencrypted copy hitting the disk on first startup)
21:29 esotericnonsense but that one seems to make securing electrum impossible unless there some magic c extension stuff