Transcript for #bitcoin-dev 2017/09/05

06:24 waxwing i think if anything this place is way underused; you can't ask general dev. or tech. questions in core-dev, and you can in #bitcoin, but that place is often a madhouse.
06:24 waxwing people do still ask things here, but not that much.
11:59 cad Does bitcoin miners check transaction's senders balance?
11:59 cad before accepting it and including it in a block
12:01 acagman mustafa
12:02 cad efendim ahmet abi
12:02 acagman nasilsin yahu ?
12:03 ozlemT selam
12:03 acagman oo ozlem hosgeldin ya
12:03 ozlemT hosbuldum ahmet abi
12:04 acagman ooo ilker sende gelmisin
12:04 ilkmccr acagman: saygilar abi
12:23 Emcy cad yes but not int he way you probably think
12:23 Emcy also there are no 'balances'
12:48 cad Emcy: I know that there are no 'balances'. What I meant was does the miner runs detailed checks on the transactions before including them in a block and if it didn't (e.g. an evil miner), will the network mitigate such case? If yes how?
13:24 dviola hi
13:24 dviola I just saw this in my debug.log, any ideas what it means? 2017-09-05 13:19:06 ProcessMessages(version, 113 bytes) FAILED peer=14
13:26 dviola is it something critical?
13:33 dviola it's syncing fine but I saw this in a few lines in the debug.log
13:34 Emcy just a duff peer
13:35 Emcy cad miners are incentivised to check what they include in a block is valid
13:35 Emcy if its not, the rest of the network will orphan the block when the miner releases it if it has an invalid tx and that will cost mr miner like 60,000usd
13:36 dviola I see, thanks
13:36 Emcy i think at one point some miners were doing validationless mining while trying to chose down the orphan rate, but thats not so common anymore
13:36 dviola is it safe to ignore?
13:37 dviola I guess it is
13:38 cad Emcy: thx
13:38 acagman saol gardas..
13:39 Emcy i think antpool accidentally made an invalid mis-ordered block one while totally not doing asicboost kek
13:41 Emcy dviola bitcoin will just ban peers that keep sending nonsense
13:42 dviola Emcy: good, so this was that basically?
13:43 Emcy look at the banscore
13:44 ozlemT #namecoin-dev
13:44 dviola Emcy: how?
13:44 dviola I'm using bitcoind
13:55 dviola oh, I see now
13:55 dviola just started qt
13:58 molz dviola, "bitcoin-cli listbanned"
13:59 dviola oh, thanks
14:02 dviola I get an empty []
14:02 dviola maybe because I restarted bitcoind?
14:03 molz does your log say your node banned some IP?
14:05 dviola no
14:09 dviola I see 'FAILED' as in: 2017-09-05 13:19:06 ProcessMessages(version, 113 bytes) FAILED peer=14 with a few peers
14:09 dviola but no bans
15:21 dviola so if this peer is sending me crap and I see a FAILED, why I don't see it as banned?
15:26 dviola or are there some cases where core will not ban them?
15:33 molz dviola, do you see anything that says "ERRORS" after that line?
15:33 dviola molz: no
15:35 molz dviola, and your node is not stuck? it's still downloading blocks, correct?
15:35 dviola correct
15:35 dviola it's not stuck
15:36 dviola I think it will finish syncing tomorrow
15:36 molz ok, i wouldn't worry about it, probably something not important
15:37 dviola ok, thanks
16:48 boltzar I am using bitcoind json rpc api to send bitcoin to multiple bitcoin addresses. I am using sendrawtransaction. When i have 24 unconfirmed payments sent from my account, i can't send anymore because i get this error : 64: too-long-mempool-chain . I have increased the maxmempool to 1500 but it still doesn't fix it. Any ideas on how to fix it ? I want to be able to create for example 300
16:48 boltzar unconfirmed transactions and not to receive that error.
16:48 boltzar Hey!
16:50 ossifrage boltzar, are all your transactions from the same address?
16:50 boltzar yes.
16:51 ossifrage So they are a single chain of unconfirmed transactions? Having a limit for that sounds like a sane thing to do
16:53 ossifrage You would be much better off having multiple outputs in fewer transactions then a long chain like that
16:54 boltzar ossifrage i can't have multiple outputs because my business is not like that...
16:54 boltzar orders come for example at 1-2-5-10 minutes apart from each other
16:55 ossifrage Well, you can either pay larger fees, or you can check if the transactions are getting backed up and do a CPFP transaction
16:56 boltzar CPFP ? sorry, i don't know what that is...
16:57 ossifrage child pays for parent... at the end of your chain of transactions, before you hit the bitcoind limit, just do a transaction with a larger fee to cover whatever part of the chain hasn't made it into a block
16:58 ossifrage Or you could just recompile bitcoind with a larger chain transaction limit, but you will have to find someone else to tell you if that is a really bad idea
17:00 boltzar i have tried now something. after i got "64: too-long-mempool-chain" i made a transaction with a higher fee, and i still couldn't make the payment
17:01 ossifrage boltzar, you would have to do that before the limit kicks in
17:01 boltzar if i do that before the limit kicks it, do i need to wait for that transaction to get confirmed also ?
17:01 boltzar because i can't wait.
17:02 ossifrage That long chain of (I'd assume low fee) transactions isn't overly appealing to the miners, they are much more likely to pick other smaller transactions then your long chain
17:02 boltzar when somebody pays me, my api has to send instant the bitcoin and as we know there are some days when you can wait for hours to get confirmations
17:02 boltzar well i use segwit at the moment
17:03 ossifrage First thing you can do is split your coins over multiple addresses, that will help this specific problem, I'm not sure it is wise to bump the long chain limit
17:04 boltzar can you tell me the reason for this ? why it's not wise ?
17:05 ossifrage Because you are decreasing your chances of getting in a block. All those transactions have to go into blocks in the right order because they are a dependency chain
17:07 boltzar before i made my own json api i used the one from block.io . I had times when i had 200 unconfirmed transactions . They eventually all got confirmed, a bit late, but they got confirmed. I want to make like that...It's not a problem if they get confirmed a bit late...
17:07 ossifrage Are you sure they all came from the same address chain?
17:08 boltzar Yes.
17:08 ossifrage It looks like there are command line options to control this for bitcoind
17:09 boltzar that's what i'm trying to find out..
17:09 ossifrage I think the thing you want to change is "-limitancestorcount", the default is 25
17:09 arubi that's only local policy. changing that won't make other nodes relay that long chain
17:10 boltzar i think this also ?
17:10 boltzar walletrejectlongchains
17:11 ossifrage arubi has a good point, you might be shooting yourself in the foot making that change (I'm assuming it is some sort of anti-spam protection)
17:13 boltzar what do you mean by this ?
17:13 arubi that and if you get to a point in time where you have a long chain which isn't confirmed, you're better off just putting all those outputs in a single transaction..
17:14 boltzar It happens very rare to have 200 unconfirmed transactions
17:14 ossifrage arubi, his point is that the transactions are happening over time, not batched
17:14 boltzar yes, exactly
17:15 boltzar and i have to instantly send the bitcoins to the clients
17:15 arubi yea I understand, bitcoin isn't very good for long chains of transactions, plus it takes about 10 minutes between blocks
17:15 ossifrage boltzar, the first thing to do is to slowly bump your fee as the unconfirmed chain gets larger and potentially use multiple addresses
17:15 arubi yep, just prepare many outputs that you can use
17:15 arubi take one big input, send it to 1000 addresses of yours
17:16 boltzar do you guys have any tip for me how to calculate the fee ?
17:16 arubi every time you need to pay, grab one of the 1000 that are already confirmed. eventually you can just consolidate small outputs at the lowest level
17:16 ossifrage But if they other nodes aren't going to forward long chains of unconfirmed transactions, then it doesn't help you to make the transaction right away if your customers aren't going to see it
17:16 boltzar i have searched everywhere but to be honest didn't find anything i can use
17:16 boltzar they all give me very big fees.
17:17 arubi you can just wait for v0.15.0 . it'll have proper fee estimator
17:17 arubi or use rc3 which is out now. I've been running rc1 for a while and it worked well. updating to rc3 now
17:17 ossifrage no need to wait... the 0.15.0 fee estimator is better, but it still overpays fee wise
17:17 boltzar for example i use segwit now and my fees are 0.3 $
17:18 arubi not sure what the fees are now. I just always underpay and wait :)
17:18 arubi got a 1 satoshi/byte tx confirmed over the weekend. all is fine
17:18 boltzar sorry for asking arubi, i'm not a pro at this ... what is rc3 ?
17:19 arubi release candidate 3
17:19 boltzar Fee per byte 27.903 sat/B
17:19 boltzar this is what i have now
17:19 arubi doesn't sound so bad
17:19 ossifrage My last 1 sat/byte transaction took over 20 days to confirm due to the recent mempool constipation
17:19 boltzar :))
17:19 boltzar one more questions if you can figure it out
17:19 boltzar i never understood this
17:19 arubi yea 1/b is pretty extreme, but fun to do once in a while if you're sending to yourself anyway hehe
17:20 boltzar check this tx out : 6938726463d5a3ab25de805538ad591043258d814544317512cfe0046a6c33c7
17:20 boltzar on blockchain.info it doesn't show
17:20 boltzar on chain.so it appears
17:20 boltzar on blockr.io it doesn't appear
17:20 ossifrage boltzar, that is fee is pretty high, should be able to go much lower only have to wait a few blocks (right now) https://jochen-hoenicke.de/queue/#2h
17:20 boltzar i had this issue for a long long time with some of my transactions. what's the problem ?
17:21 arubi probably because they're one of the very long chain..?
17:21 boltzar i understand. so that's the issue.
17:21 arubi right, block explorers might be running with the same default rules at 25 max
17:22 boltzar for example if i have 5 segwit addresses in my wallet
17:22 boltzar when i use createraw..sign...send... does it take randomly from one of those addreses ?
17:22 arubi no, it uses the one from the input
17:22 boltzar blockchain.info i know for sure it has 25 max. :) i used the api from them and the same issue i had with their api also
17:23 ossifrage That looks like ~23 unconfirmed transactions on that address, right at the 25 txn limit
17:23 boltzar ossifrage i had 25 unconfirmed, 2 got confirmed.
17:23 boltzar i was just making tests
17:23 boltzar to figure this out
17:24 ossifrage boltzar, FYI, if I was writing a spam detector those transactions would be labeled as spam
17:25 ossifrage same input, same outputs, many transactions all created at the same time == spam
17:25 boltzar to be honest i'm kind of crazy ... i use the same address just to keep track of my balance easy.. i have my bitcoin address on watch on the blockchain api on my phone
17:25 arubi that's not very good for your customers
17:26 boltzar yes, but that was just for test.
17:26 ossifrage Once those clear out, try your test again with more realistic timing and output addresses
17:26 arubi but if one customer gets a transaction from you, they can then track your address too and learn about your business
17:27 boltzar i understand that risk. another issue was segwit...to move my change to different segwit wallets...it's kind of difficult for me
17:28 boltzar sendtoaddress moves the funds around, but not to segwit addreses
17:28 arubi sendtoaddress can send to p2sh addresses
17:29 arubi and that's most of what people use now as segwit addresses, so really not sure why it couldn't send for you
17:29 boltzar i mean for example i have 1 bitcoin. 0.5 goes to the customer + fee and 0.5 has to return to me. that return address is not segwit
17:29 arubi ohh
17:29 arubi yea, that's right.
17:29 boltzar and next time i send...i send from non-segwit
17:29 arubi why aren't you using fundrawtransaction?
17:30 boltzar so to move funds around with createraw...difficult for my brain.
17:30 arubi or are you?
17:30 boltzar no. create, sign, send
17:30 arubi alright, so if you place fundraw between send and sign, you can set a change address
17:30 boltzar don't know what's the difference. to be honest i have 1 week since i'm playing with bitcoind
17:30 arubi you place the output from createraw into fundraw, then the output from fundraw into signraw
17:31 boltzar you can set a change address on sendraw also
17:31 arubi sendraw? no
17:31 arubi sendraw just sends signed and complete transactions
17:31 boltzar on createraw i have to put the change address
17:31 arubi better not
17:31 boltzar then sign and send
17:32 arubi better let fundraw deal with it. it will also select an input, then you put that into signraw, then sendraw
17:32 boltzar fundraw selects it's own change address ?
17:32 boltzar i don't have to define it ?
17:32 arubi yes
17:32 boltzar but it won't select a segwit address :)
17:32 boltzar same thing like on sendtoaddress
17:33 arubi it won't but you can set one
17:33 arubi if you set one in createraw, your signraw won't know that it's a change address and just add a new one
17:33 arubi so the only way to specify one yourself right now, is using fundraw
17:34 boltzar i have to try this
17:34 boltzar it sounds good. if it would add a new segwit address, it would be perfect
17:35 arubi sure. just one thing, the wallet can't restore these addresses in case you need to restore from backup. you need to add them all back manually yourself, so not sure if you should use it with lots of money right now
17:36 arubi I mean, it's not that these are completely lost in case the wallet is lost. they can always be restored, but it's a manual process now
17:36 boltzar will make some tests with 10-20 $ on it :)
17:36 arubi nice :)
17:36 arubi (you can also use testnet)
17:37 boltzar you're awesome dude. i've been with my head into this api for a week. it's just awesome, but difficult.
17:37 arubi cheers man, it's great knowing people are having fun with bitcoin stuff
17:37 boltzar so , related to the fee. how much do you recommend for me to use ? like i said, i use 0.3 $ = 0.0000692
17:38 boltzar what's your suggestion ?
17:38 arubi you can use https://jochen-hoenicke.de/queue/#2h for looking at what other people are paying
17:38 ossifrage boltzar, the right fee always depends on the current state of the network, using a fixed value is just wasting money
17:38 arubi so just set to whatever lowest color is getting cleared :)
17:41 boltzar i'm dumb as f*ck
17:41 boltzar don't know how to read this graph
17:41 arubi it's got the legend on the left, bars are at 1mb intervals
17:42 arubi at each block, a bunch of transactions get into a block, you can see the graph drop when that happens
17:42 arubi usually higher paying txs get into a block first, then as more clears over time, lower fees do
17:43 arubi but right now you can see that two or even one block will clear the current peak, so a smart miner will just include all transactions in that peak
17:43 arubi which means a high fee paying one has the same chance to get into the next block as a low fee paying one
17:44 arubi er, the website is https://core.jochen-hoenicke.de/queue/#24h
17:44 arubi (change the view to 2 hours, not 24 hours)
17:45 boltzar at the pending transactions for example there is 600+ : 0.024 btc
17:45 boltzar 0.024 / 600 ?
17:45 arubi 600 is what?
17:45 boltzar sat/b
17:47 arubi wait, you paid 0.024 btc for a 600 byte transaction?
17:47 boltzar no no. i was reading the graph
17:48 arubi ohh, yea, someone is paying very expensive fees and losing a ton of money
17:49 arubi it means they're paying 4000 satoshi / byte
17:49 boltzar to understand better, for a 248 (bytes) what's a good transaction fee ?
17:50 boltzar 5 satoshi / byte ?
17:50 arubi right now it is
17:50 boltzar 10 ?
17:50 arubi it doesn't matter how big the tx is
17:50 arubi you're paying by the byte
17:50 arubi boltzar, how much does it cost to ship 1 2x2x2 box?
17:51 arubi you pay per weight, the size nearly doesn't matter
17:51 boltzar Weight 662
17:53 arubi so the setting in core is "pay this much BTC per kilobyte" right
17:53 boltzar yes
17:53 arubi 1 bitcoin is 100,000,000 satoshis. it's asking you how many satoshis per 1000 kilobytes
17:54 arubi so basically the setting is (5 * 1000 / 100000000)
17:54 arubi ;;calc 5 * 1000 / 100000000
17:54 gribble 5e-05
17:55 boltzar 0.005 / kb right ?
17:55 arubi 0.00005000
17:55 boltzar sorry
17:55 boltzar yeah
17:56 boltzar 0.00003 aprox for that 662 bytes transaction
17:56 boltzar ?
17:57 arubi yea, just over that
17:57 arubi 0.00003310 specifically. fundrawt has a 'feeRate' field you can set at funding time if the preferred fee changes
17:58 boltzar yeah, unfortunetly i have to calculate that in php
17:58 boltzar i have to make it auto
17:58 arubi well no, because you put 0.00005000 in there, not the end result :)
17:58 arubi it'll include 0.00003310 when you send a 662 byte tx
17:58 arubi and whatever else if it's a different size
17:59 boltzar i mean for each transaction, i have to calculate it in php. because when i create the raw transaction, i put the amount to send the fee and the return amount.
17:59 arubi but you don't if you use fundraw, that the point
17:59 boltzar if i put that fee, it will work only on sendtoaddress ...
17:59 arubi you don't have to*
17:59 arubi bitcoind will take care of that for you
17:59 boltzar i see
17:59 boltzar interesting
18:00 boltzar i have to lookup this function asap
18:00 arubi takes a simple json with whatever you want to set. probably changeAddress and feeRate
18:00 arubi (and the hex from createraw)
18:01 arubi the hex from createraw should not contain an input of yours already, or a change address. just the output to the customer
18:01 arubi fundraw will choose inputs to cover the output, set the fee and the change, and then you get a complete unsigned transaction that you can pass to signraw
18:02 boltzar very nice. i will empty the wallet and try this function
18:02 arubi cheers. I'm off for a while. good luck
18:02 boltzar limitancestorcount i have to put into bitcoin.conf ?
18:02 arubi nah, don't set it at all
18:02 arubi other nodes don't have it set, so it's not like it's going to help you
18:03 boltzar i want to set it , because i will have problems with my orders...
18:03 arubi you'll still have the same problems
18:03 arubi setting it won't change the end result of long chains not being relayed
18:03 boltzar how come when i used block.io i sent 200-300 payments
18:03 boltzar all unconfirmed ?
18:03 arubi that's because they can't relay these to a miner for you
18:04 arubi the network largely ignores them
18:04 arubi because of that same policy
18:04 boltzar the payment went thru...
18:04 arubi when it's a 23 long chain, sure
18:05 boltzar really, it's not BS.
18:05 boltzar i've used them 1 year
18:05 arubi anyway, gotta go. you can set that in your bitcoin.conf no problem. but it might just cause your software to pay and pay without end
18:06 ossifrage It is possible that the end of the chain does not show up in other nodes until the head of the chain ends up in blocks (so it will work eventually)
18:06 boltzar arubi, thanks a lot !
18:06 boltzar ossifrage yes you are right. not all the transactions showed up on the blockchain, untill they got confirmed
18:07 boltzar they showed up only on chain.so , but just there
18:07 ossifrage So block.io might just crank up those limits and let the network sort it out...
18:08 ossifrage But it won't give your users the feedback they expect if the transactions are getting filtered by nodes
18:09 boltzar i know, it was frustrating, but i just want to find out what's the solution. last week i didn't even know that the output of bitcoin has a return address for the rest of the funds :))
18:09 boltzar i'm new at this subject
18:10 boltzar untill now i've used bitcoin api's build by other websites and i'm tired of those issues and i want to create my own. that's why i want to know what's the best solution.
18:11 boltzar i will try with fundraw to see if it still uses segwit on the return address and if yes, i will fund 20-30 addresses so i won't have problems with the long chain
18:18 boltzar anyway. thanks a lot ossifrage and arubi. you guys are awesome
18:18 arubi o/
19:57 JackH Can we get an RPC to see data usage in Bitcoin? Would it make sense?
19:59 arubi might be that one of the debug options tracks it?
19:59 arubi I'm not sure, I never checked.
20:00 JackH I checked the RPC calls, nothing for tracking data usage
20:02 JackH I think it can become useful for certain nodes to keep track of data usage. I just did 91GB in 3 days
20:02 arubi so -debug=<category> doesn't show it for category 'net' ?
20:03 JackH huh?
20:04 arubi bitcoind's debug flag, might track network usage
20:04 arubi I mean, I'm asking if you tried it, because I didn't
20:05 arubi output should appear in debug.log
20:05 JackH I never tried this to be honest
20:05 arubi I'm gonna try
20:06 arubi well it does tell me whatever the bytes for each message sent \ received
20:06 arubi so I guess it does just that ;)
20:06 JackH wait so how does it store this?
20:06 JackH whats the output?
20:06 arubi in debug.log
20:06 arubi 2017-09-05 20:06:16 initial getheaders (483711) to peer=6 (startheight:483712)
20:06 arubi 2017-09-05 20:06:16 sending getheaders (997 bytes) peer=6
20:07 arubi pretty sweet
20:07 JackH but the output of 10GB will be impossible to keep track of like this
20:08 arubi max message is that many right
20:08 arubi and really there aren't messages bigger than one block
20:08 arubi so, 4mb max
20:08 arubi you can't keep track of gigabytes without tracking bytes
20:08 arubi you just accumulate :)
20:09 JackH there must be something in the GUI then that counts all the bytes and draws the graph and also keeps track of the data per session
20:10 arubi sounds pretty useful really. I made some output for myself of connected peers and the amount of data sent \ received from me to them
20:11 JackH do you read from the debug then?
20:11 JackH or how do you grab the data?
20:12 arubi no, just rpc calls and prettyprinting in bash
20:12 JackH ahh
20:12 arubi https://gist.github.com/fivepiece/07fdaa7bce38ffa0c77bdb5c40a69a25
20:13 arubi I run that with `watch -d -n15 ./peerinf.sh`
20:13 arubi it's somewhat buggy when nodes have spaces in their useragent :\
20:13 JackH ah well that can be cut out
20:13 JackH doesnt matter (for me) who downloads what
20:14 JackH as long as I have overview of total consumption
20:14 arubi getpeerinfo has all the details, but data disappears when a peer disconnects
20:14 arubi it's just for real time stats
20:15 arubi /dinner
20:21 tloriato_ Hello. I was wondering about HD wallets creation. I would like to generate deterministics public addresses linked to a specific BIP39 24 word list, but without having to expose the words. I came across a specific implementation that can use BIP39 to generate BIP32 addresses (bitcoinjs lib) and it does that using the BIP39 seed and not the words. Is the seed a "disposable" information?
20:22 tloriato_ Disposable in the sense that if a hacker manages to grab that information it would not be harmful in any way?
20:22 tloriato_ (monetarily speaking)
20:39 tloriato_ Ok... I that it's actually quite important, in the sense that I was able to derivate the BIP32 Root Key from the BIP39 Seed and then get the private keys became trivial. Does anyone knows how I could achive what I'm aiming?
20:51 arubi tloriato_, it sounds like you didn't read bip32 or bip39 at all? there's info in bip32 specifically about public derivation paths
20:53 tloriato_ arubi , I read it but I was looking for an implementation of the described process in the "Public parent key → public child key"
20:54 tloriato_ I believe that would cover my use? Being able to generate new deterministics public address on a server without having to keep any sensible data on it?
21:01 arubi yea but, the process is right there in bip32, and there are links to implementations at the bottom
21:12 tloriato_ yes, you are right. i'm currently looking into the JS implementation to see if I can pinpoint where he implemented the desired Public parent key → public child key process
21:12 tloriato_ but have been unlucky
21:14 arubi look for parts where it does hmac-sha512
21:15 arubi one of them will be public child key derivation
21:17 tloriato_ thank you arubi
21:17 tloriato_ i think that i found it https://github.com/sarchar/brainwallet.github.com/blob/bip32/js/bip32.js#L229
21:18 arubi yea that's it
21:18 tloriato_ so it's possible to do what i'm looking for, right? generate a public address from another "master" pub address (linked to a bip39 passphrare)?
21:18 tloriato_ just to be clear, sorry
21:19 arubi it is, but you have to remember to never export a private key from that path
21:19 tloriato_ (without knowing sensible information, so i can keep them off computers)
21:19 tloriato_ uhm...
21:19 arubi you can give someone a master public key, and they can generate addresses from it
21:19 arubi but if you also give them one single private key from one of these addresses, then they can get all other keys
21:20 tloriato_ oh, i see!
21:20 tloriato_ that's incredible and exactly what i've been looking for
21:20 tloriato_ i'll see if i can find it in the bitcoinjs lib to use easily, otherwise i'll implement it there
21:20 tloriato_ thank you!
21:20 arubi yw
21:53 jimpo Is it known or is there a leading theory why Satoshi used double SHA256?
23:03 Emcy in case of a marginal sha256 break maybe?
23:03 Emcy like what happened with sha1
23:04 Emcy you know like how DES got weakened significantly and everyone said screw it DES but 3 times
23:57 esotericnonsense double sha protects against length extension
23:57 esotericnonsense i'm not aware of anything in bitcoin that needs that particular property though