Transcript for #bitcoin-dev 2013/03/11

00:00 HM Luke-Jr: lol, not bitcoind rpc's
00:00 HM this is something i'm working on
00:03 sipa Luke-Jr: what % of 0.8 nodes do you see?
00:55 Luke-Jr sipa: 38%
00:55 sipa Luke-Jr: hmm, nice
00:56 sipa i get a reachability-weighted ratio of 47% even
00:56 Luke-Jr reachability-weighted?
00:57 sipa count each node weighted by its reachability%
01:04 CodeShark might such statistics not be skewed by whether nodes accept incoming connections?
01:04 sipa i wouldn't say skewed; it's literally what i'm calculating
01:05 sipa of course it won't match the population of all nodes
01:05 sipa but there's no way to observe those anyway
01:05 warren 0.3.x nodes are still compatible?!
01:05 sipa yes
01:06 sipa there has never been a hardfork, ever
01:06 CodeShark I'm saying that even major hub nodes might not be ranked too high by a bot if, say, the hub nodes have long lists of connect=
01:06 warren ah
01:06 sipa and only one backward-incompatible network protocol change (which kicked off pre=0.2.10 nodes)
01:06 doublec sipa: p2sh didn't count as a hard fork?
01:07 sipa doublec: absolutely not
01:07 sipa that was only a softfork (=backward compatible)
01:07 doublec ok, I thought a bunch of nodes got orphaned blocks after that
01:07 sipa yes, it was a fork, but not a hard one
01:08 sipa the difference is that in the case of a soft fork, only those mining on the old chain are vulnerable, but never those trusting the longest chain
01:08 sipa while in a hardfork, every old client is dropped on the fork
01:08 doublec ah ok
01:08 sipa hence, softforks are safe as soon as a supermajority of miners upgrades
01:09 sipa hardforks are safe as soon as _everyone_ (not just miners) upgrade
01:09 doublec that makes sense, thanks
01:13 Luke-Jr sipa: there has..
01:13 warren Luke-Jr: is that nodes that are listening only?
01:13 Luke-Jr sipa: when most script opcodes were disabled, for example
01:13 Luke-Jr warren: yes
01:13 sipa Luke-Jr: softfork
01:14 Luke-Jr hmm
01:14 sipa (clients which have all opcodes enabled still, will still validate the new best chain)
01:14 Luke-Jr yeah, I guess you're right then
01:15 sipa at that point in time the difference probably didn't matter much
01:15 sipa and the OP_RETURN related bugs were probably bad enough that if a hardfork had been necessary to fix them, it wouldn't have been a problem
01:16 sipa i wonder whether the OP_RETURN change (instead of evaluating input + OP_CODESEP + output, evaluate both separately) can result in something that is now accepted but wasn't back then
01:16 sipa if that's the case, it was a hardfork, though i can't think of anything
02:03 HM "It's like playing Satoshi Dice, only you are using your bitcoins to buy something epic, supporting models, contributing to charity, and getting something in return."
02:03 64MACUB4N standard bitcoin client transaction fee are insane!
02:47 Lophie hello, anyone interested in a conversation regarding electrum
02:48 Lophie I have things I need to know and don't know where to find
02:48 Lophie and yes I tried google o_o
02:50 phantomcircuit did you try #electrum
03:31 Lophie brb
03:36 unknown45682 hi
04:34 bonks In bitcoin-qt 0.8 I tried the importprivkey command via the Debug window, and any subsequent command does sent does not return a response. Is this expected behavior?
04:35 bonks s/command does/command
04:47 bonks That was weird. So I saw it appear in my address book. Then decided to restart bitcoin-qt. It took several minutes to shut down. I started it again and the address moved from my Address book to Receive coins, where I expected it.
04:53 bonks Oh I think I understand it now, it'll just take a long time to process
04:53 bonks :x
05:04 warren bonks: I heard something about it rescanning the entire blockchain after import key (
05:08 bonks warren: Yea I just realized that in help rescan is the last parameter and defaulted to true
05:09 bonks On my linux box I did not see that
05:09 warren I think it can't know your balance if you don't rescan?
05:10 bonks Probably. I'm just discovering paper wallets and am testing it out :)
05:14 jgarzik
05:14 jgarzik "Confirmation time vs price: Another reason to keep the blocksize where it is? #bitcoin
05:17 warren What does he mean by "slower miners"?
05:49 CodeShark dunno - not sure that part really makes sense
05:49 CodeShark the time to include new transactions (including the network latency until the miner sees the transaction) is negligible in comparison with the time it takes to mine a block
05:51 warren supposedly including more tx's increases your chances of orphans, but I haven't seen the math of that
05:53 CodeShark you mean deadend sidechains?
05:53 CodeShark I hate the usage of the term "orphan" to refer to a sidechain since this usage has absolutely nothing to do with the way the term is normally understood in the English language - which is something without parents
05:54 CodeShark and we absolutely need a term to refer to blocks that don't have known parents
05:54 CodeShark and the term "orphan" is perfect :)
05:54 sivu bastards instead?
05:55 CodeShark heh, that's not bad
05:55 sivu for the sidechain
05:55 CodeShark I've also heard "extinct" proposed
05:55 CodeShark an extinct block is a block which nobody mines against anymore
05:55 CodeShark which is not in the main chain
05:56 sivu thats more like evolutionary misstep than extinction because the block still exists
05:56 CodeShark yeah, true
05:56 CodeShark and in principle it could be brought back to life (with sufficient computational power)
05:58 CodeShark bastard block isn't that bad :)
05:58 CodeShark lol
06:00 CodeShark anyhow, getting back to warren's comment - how would including more tx's increase chances of bastard blocks? I suppose if two blocks are mined at around the same time and a miner sees both he might choose to mine against the smaller block so he can pocket fees from transactions in the other
06:02 CodeShark also, sending a larger block takes slightly longer so some might argue it propagates a little more slowly (although I doubt this is significant)
06:02 weex it takes longer for each relaying node to validate blocks with more transactions in them
06:02 weex so network propogation is slower
06:02 CodeShark yeah, but by how much?
06:02 CodeShark can't be more than the order of seconds
06:03 weex i think gavin calculated this in a recent foundation or forum post
06:03 warren CodeShark: bigger blocks take longer to propagate, increasing the likelihood of a competing block to orphan it? I'm not really sure.
06:03 CodeShark in any case, it's far less than the 10 expected minutes between blocks
06:04 warren CodeShark: apparently some pools reduced tx's to reduce orphans
06:04 CodeShark I guess we need a new word to describe blocks without known parents :(
06:04 CodeShark because we're going to have nomenclature confusion forever
06:05 weex vestigal
06:07 CodeShark vestigal is a great term for deadend sidechain blocks
06:08 CodeShark they still exist...but they no longer support the organism
06:09 CodeShark I'm gonna see if I can convince piuk to change the term on
06:10 CodeShark I don't mind using a different word for these blocks - the problem is that now I don't have a word to describe blocks without known parents
06:11 CodeShark and the term orphan to mean "an object without known parents" is consistent with the way satoshi used it and the way it's used in the bitcoind source code
06:12 CodeShark and inconsistent with the way uses it
06:13 warren CodeShark: Folks need to get over "the way satoshi used it" as reason for anything.
06:13 warren CodeShark: He is wicked smart, but he can't have got everything right.
06:14 CodeShark that's not the main reason - the main reason is that that's what it actually means in the English language
06:14 CodeShark lol
06:14 CodeShark that's how it's construed in the English language
06:14 CodeShark whereas's usage is totally foreign
06:14 warren The parents disowned it, or the parents were killed.
06:15 CodeShark there are many things that satoshi did that I disagree with - but his usage of this term is not one of them :)
06:17 CodeShark someone whose parents are killed is not necessarily an orphan
06:17 CodeShark and a parent block has no say as to whether its children are disowned
06:17 CodeShark it's the later generations that determine whether it stays in the main chain
06:18 CodeShark furthermore, an orphan's parents might still be alive
06:18 CodeShark orphan just means we don't know who the parents are
06:18 CodeShark anyhow, it's silly to argue this :p
06:20 CodeShark if everyone wants to redefine "orphan" to mean a block in a chain which is no longer being extended, I don't really have a problem with that - as long as we also have an unambiguous term to mean a block whose parents we don't know...and make sure to mention this in the documentation so people familiar with typical English usage of these terms doesn't get confused
06:22 CodeShark also, comments should be added to the bitcoind source code to point out this discrepancy :)
06:23 CodeShark I guess we could use the term "unconnected chain"
06:23 CodeShark rather than "orphaned chain"
06:23 CodeShark it might still become a side chain if we learn of the blocks connecting it with the genesis block
06:24 CodeShark so "orphaned" status isn't necessarily permanent
06:29 CodeShark or the term "isolated" might also work
06:29 CodeShark an isolated block - or an isolated chain
06:30 warren isolated has an incorrect implication of valid
06:31 CodeShark island block and island chain?
06:31 CodeShark lol
06:33 CodeShark problem with the term "unconnected" is that we're only referring to backwards connections - it might still be referenced by future blocks
06:33 CodeShark unlinked?
06:33 warren by future blocks that could reorg the entire chain and become the valid one?
06:34 CodeShark the block chain is a singly linked list - what's the official term for an element in a linked list which has a bad pointer? :)
06:34 CodeShark or for a linked list which contains a node with a bad pointer
06:37 CodeShark "unconnected" is analogous in this context with the way it is used to refer to transactions which have bad inputs
06:38 CodeShark or inputs whose scripts we cannot validate for whatever reason
06:40 CodeShark anyhow, I don't mean to be pedantic - there are real practical reasons for having an unambiguous term
06:40 CodeShark :p
06:42 warren CodeShark: you're having a long conversation with yourself
06:43 CodeShark anyhow, back to the idea that including more transactions increases the risk of an abandoned block
06:43 CodeShark (I like the term abandoned, btw)
06:45 CodeShark isn't it really the number of inputs that determines how quickly it can be validated?
06:45 CodeShark not the number of transactions
06:46 rdponticelli1 I'm getting depressed... All those poor little blocks... :P
06:46 CodeShark lol
06:46 warren CodeShark: perhaps "unfavored"
06:47 CodeShark the bottleneck in transaction validation is signature verification - which happens once per txin
06:47 CodeShark or more, perhaps
06:48 CodeShark in the case of a multisig
06:51 CodeShark another thing about block validation - nodes can validate transactions prior to seeing them in blocks - and there's no real need to revalidate the transaction when the block is seen
06:53 CodeShark although I don't think bitcoind does this optimization
06:58 CodeShark or nevermind...
06:59 CodeShark once a transaction is marked as connected it is no longer validated
06:59 CodeShark (at least the signatures aren't verified again)
07:01 CodeShark so this would only apply to transactions in the block that are not already in the mempool
08:48 moarrr Bitcoins @ 5JzdYNEKBZhCmubuhosCWsM5e6cDguhsdnSK7jumegQU9cZBKq3 (1jqfg3fCu4ssKWANk2FvXoU3x54pBPGoc)
08:58 CodeShark current balance is 0 :p
08:59 CodeShark and it never had an output large enough to even be worth spending
09:05 CodeShark lol
09:08 lianj wonder which one makes it
09:18 lianj ah, the higher fee one ofcourse
11:18 whitez Hello. Super n00b question, but where can I download bitcoind?
11:18 sipa
11:19 kriqCoin I hear there was a meetup in Zurich of BC devs
11:19 kriqCoin whats up with that?
11:21 epscy they were discussing the killswitch
11:21 whitez sipa: Is it embedded in Bitcoin-Qt?
11:25 CodeShark Bitcoin-Qt is essentially bitcoind with a GUI on top of it
11:26 whitez CodeShark: Ok, I see. I
11:26 whitez I've downloaded Bitcoin, so I'll try to load it as a daemon
11:27 CodeShark you can also build it yourself if you prefer from the github repository
11:28 CodeShark and you can also get it as a package with several linux distros
11:35 whitez CodeShark: It works!!!
11:35 CodeShark :)
11:35 whitez Magic :D
11:35 whitez And the icon is all nice and green. Awesome. Thanks very much! ???
11:36 sipa whitez: if you're seeing an icon, you're not running bitcoind
11:36 TD kriqCoin: yeah
11:36 TD kriqCoin:
11:38 CodeShark I'd also be interested in what stephan has to say :)
11:38 Scrat google performing a 51% attack in the zurich meetup?
11:41 whitez sipa: Do you know how to make it not-an-icon ?
11:41 whitez "Bitcoin-Qt -datadir=1 -daemon"
11:41 sipa whitez: bitcoin-qt is the GUI, bitcoind has no GUI, so if you see any GUI, you
11:41 sipa 're not running bitcoind but bitcoin-qt
11:41 CodeShark whitez, if it's built as Bitcoin-Qt you cannot make it headless - you need to build it headless
11:42 CodeShark so you need to either get a bitcoind binary or build it from source
11:42 whitez Ah, I see - build from source. Gotcha. Will do. Thanks. :)
11:46 CodeShark Bitcoin-Qt and bitcoind are two different builds that use the same underlying source code for the daemon portion
11:47 whitez I'm on a Mac, so working off this now:
11:48 sipa oh, right; for windows and linux, bitcoind is just bundled with the package, but on OSX it's only bitcoin-qt
11:49 CodeShark whitez, you can also install macports and then port install boost
11:49 whitez boost?
11:49 CodeShark it's one of the library dependencies
11:50 CodeShark
11:50 whitez It's whirring and clicking now....
11:50 CodeShark what version of OS X are you running?
11:51 whitez CodeShark: Oh, that's helpful, thanks. 10.8.2
11:51 CodeShark then it's the same as for 10.7
11:53 whitez I started by trying "sudo port install bitcoin", I don't need a super up-to-date version.
11:53 CodeShark is there a macport for bitcoin? lol
11:54 sipa whitez: you really want an up-to-date version - old versions have significantly worse performance, and often security bugs
11:54 CodeShark and are also likely to be abandoned by the network in future releases
11:54 whitez sipa: Oh, ok. I'll let this finish and then install a shiney new version
11:55 CodeShark that build-osx.txt gives you step-by-step instructions for installing the newest release
11:56 whitez Hah! "Error: Processing of port bitcoin failed"
11:56 whitez I'll try your way now. :)
11:57 CodeShark I'd be curious as to why it failed
11:58 whitez CodeShark: I can send you the log if you want.
11:58 CodeShark yes, please do so
11:59 gavinandresen whitez: Ignore the "uncomment build_arch" instruction
11:59 gavinandresen .. that'll just get you a bunch of conflicts with your existing ports
11:59 whitez gavinandresen: Ok
12:00 kriqCoin Thanks TD
12:01 kriqCoin I am bringing along some friends)
12:56 ItsDom So satoshi client tracks up 16k known nodes. Does anyone know the justification for the number 16k?
12:57 HM it's higher than 10k and lower than 20k
12:57 ItsDom aaah, I hadn't thought of that.... ??_?? :P
12:58 ItsDom but seriously though, if only keeping about 10 live connections, why hold as much as 16k? and if network is well of 16k, why only hold 16k? Is it just a trade off between resources and network knowledge?
12:58 ItsDom well over*
12:58 HM is it actually 16k, or 16384?
12:58 davout ItsDom: why do you say it keeps only 10 connections ?
12:59 sipa ItsDom: have you read the comments in addrman.h ?
12:59 ItsDom I can't remember the exact number, but it only maintains a small number of connections, around 10 I thought. That's what I was told on here yesterday. I read the comments but it doesn't justify the numbers
12:59 ItsDom it gives the numbers, but no reason as to why (not that I could find)
13:00 davout I believe you can change it in bitcoin.conf
13:00 davout but as for the why it's a good question
13:00 gmaxwell ItsDom: It makes 8 outbound connections.
13:00 gmaxwell davout: go read the extensive comments in addrman.h
13:00 ItsDom but my original point still stands: why hold so many known nodes if you're only going to use a small fraction of them?
13:01 ItsDom I could understand it hold ALL the known nodes in the network (that would obvs be a bit silly from scalability point of view) but 16k just seems a bit....random?
13:01 gmaxwell ItsDom: You are connected to 8 outbound at any one time. But your memory is persistent.
13:01 gmaxwell ItsDom: there are potentially 2^128 possible nodes that you can be told about. You are not going to store all you've been told about.
13:02 ItsDom exactly, it would be silly.
13:02 gmaxwell No. Not silly. Not possible.
13:02 ItsDom but why not just hold, say, 512 known nodes?
13:02 gmaxwell It would be nice to store all of them??? but we used to, but then its a DOS vector.
13:03 gmaxwell ItsDom: because we need to resist a malicious peer flushing out the memory.
13:03 gmaxwell This is explained in addrman.h.
13:03 ItsDom I'll go take another look through addrman.h - I didn't notice that bit.
13:07 ItsDom Ah, I see the comment, I didn't make the link in my head the ability to fill the nodes list with malicious entries and the size of the known nodes list.
13:07 ItsDom so is this based on the assumption that an attacker wont have more than 16k compromised nodes....?
13:08 gmaxwell ::sigh::
13:08 gmaxwell Do you see the number "16k" in there?
13:12 ItsDom No, i was assuming 16k from 256 buckets for new addresses and 64 entries per bucket.....
13:15 ItsDom it's actually less than 16k though due to duplications.
13:16 gmaxwell The parameters are selected to give resistance to attack, the total is just falls out of those. There are actually 20480 slots in total, but as you note there can be some duplication (which is intended and good as it makes flushing harder). But the total numbers aren't so interesting, they're just a result of trading off attack resistance against storage.
13:19 ItsDom Aaah, thanks for the response. I suspected as much. Thanks again for taking the time to help - I really appreciate it.
14:08 jgarzik ACTION wonders what happened to [tycho]
14:08 jgarzik doesn't seem to login much anymore
14:08 gribble Error: "Tycho" is not a valid command.
14:08 jgarzik ;;seen [Tycho]
14:08 jgarzik ;;seen \\[Tycho\\]
14:08 gribble Error: "Tycho\\" is not a valid command.
14:09 gribble I have not seen tycho.
14:09 kinlo ;;seen tycho
14:09 kinlo ofcourse, that's useless
14:09 abracadabra !seen [Tycho]
14:09 gribble Error: "Tycho" is not a valid command.
14:09 abracadabra meh
14:09 abracadabra to the logmobile!
14:09 gmaxwell I saw nanotube tell some other user how to deal with that.
14:09 kinlo the problem is that gribble wants to expand stuff :)
14:09 Scrat nickserv to the rescue
14:09 Scrat [17:09] -NickServ- Last seen : Dec 06 15:09:13 2012 (13 weeks, 3 days, 23:59:58 ago)
14:10 abracadabra tycho is hard at work adding stratum to his pool
14:10 abracadabra ;)
14:10 Graet heh
14:11 gribble [tycho] was last seen in #bitcoin-dev 18 weeks, 0 days, 16 hours, 11 minutes, and 58 seconds ago: <[Tycho]> !seen tcatm
14:11 kinlo ;;seen "[tycho]"
14:11 kinlo there you go :)
14:11 kinlo rtfm still works :p
14:11 Luke-Jr jgarzik: he never did :p
14:15 gribble [Tycho] was last seen in #bitcoin-dev 18 weeks, 0 days, 16 hours, 16 minutes, and 27 seconds ago: <[Tycho]> !seen tcatm
14:15 SomeoneWeird ;;seen "[Tycho]"
14:36 WolfWindshadow_ I am working on a little coin project of my own, just to teach my kids about BTC and crypto in general, and was wondering where in the source I set the limits on total coins and the difficulty, so that I can set up my own little demo network.
14:37 WolfWindshadow_ what file would that be in?
14:37 SomeoneWeird how old are they? (just wondering)
14:37 SomeoneWeird also look at testnetinabox
14:37 _dr you're never too young to learn about generators in cyclic groups!
14:37 SomeoneWeird
14:39 jgarzik ouch, poor BTCGuild. Accidentally paid out for some generated blocks, before their server caught up with the chain
14:39 jgarzik ancient blocks, at low difficulty
14:39 WolfWindshadow_ that addy is 404
14:39 SomeoneWeird refresh a couple times, github is having some problems
14:41 WolfWindshadow_ ok... in any case, anybody know what files I would dig in to change that? I've been programming since basic on the Commodore 64 was as good as it got, just a matter of finding it....
14:42 xjrn WolfWindshadow_: asm on the c64 was as good as it got though
14:42 WolfWindshadow_ asm was a p.i.t.a. :P
14:43 WolfWindshadow_ but did freak out teachers that did not now it
14:43 WolfWindshadow_ know
14:44 TD jgarzik: oops
14:44 TD jgarzik: yeah, details like that make bitcoin programming remarkably painful and tricky to get right
14:44 xjrn WolfWindshadow_: i tuned in late, sorry, i can't be of more assistance
14:45 TD jgarzik: i've been trying to figure out how to make bitcoinj as safe an API as possible and the number of edge cases you end up with is just amazing
14:45 WolfWindshadow_ was asking where in the bitcoin source to change things to set difficulty and total coins and such to make my own small demo network for my kids
14:46 WolfWindshadow_ teaching the little cyber-monsters how to use crypto
14:46 xjrn worst-case, create some blocks on bogus datetime trajectories
14:46 TD WolfWindshadow_: hm, i was only interested in video games when i was a kid :)
14:47 TD WolfWindshadow_: the bitcoin codebase is not exactly kid friendly (or adult friendly for that matter), but look in for the constants at the top
14:48 lianj
14:48 helo WolfWindshadow_: it may be easier to teach them crypto by using bitcoin directly... there's still plenty of crypto going on even after the coin is started
14:49 gavinandresen WolfWindshadow_: BlueMatt uses a patch that sets difficulty very low as part of our pull-tester process...
14:50 gavinandresen WolfWindshadow_: ??? ah, the .patch file here:
14:50 WolfWindshadow_ thanks... code is my thing.... starting with the concepts... once I have a demo running, gonna leave it to them and their friends...
14:51 WolfWindshadow_ thanks again, and have a great day
14:57 TD sigh
14:57 TD the bitcoinj api is so crap in so many ways
14:57 gavinandresen api design is really hard
14:58 TD yeah. especially when there aren't any pre-existing examples to help you. and especially when the thing the api does is inherently concurrent. and especially when some of your users have hard latency/thread affinity requirements.
14:58 sipa the largest problem is that use cases evolve, and you don't know in what ways you'll need extensibility in the future
14:58 TD that's definitely a large problem.
14:59 TD right now one of my biggest source of bugs comes from the fact that there are listeners you can register at various points, and managing the re-entrancy and locking around that is quite hard, as basically at any listener point anything can happen
14:59 TD it's awfully handy to just have a function that runs when something happens, but it's an absolute nightmare to handle all the different possibilities.
15:00 Scrat TD: can I use the bitcoinj API if I don't run a java client?
15:00 gavinandresen api design for multithreading is hard^n ...
15:00 TD ainfo was asking me about this. i experimented with compiling it down to native code and using it from (objective) C++ a while ago
15:00 TD it worked OK. i mean it presented a basically C++-like API, with a bit of massaging
15:00 TD that was using GCJ/CNI
15:01 TD but that branch bitrotted. i upgraded bouncy castle and some code i didn't use hit a stub. i never got around to slicing the necessary parts out, as nobody seemed to care about it much
15:01 TD also, back then i was doing it for an iphone client, and then apple banned everything. so i stopped. if there's interest i can try to resurrect it.
15:01 HM Atm I'm trying to massage some APIs in to a stateless condition
15:01 HM so servers don't need to maintain any state
15:02 HM API design is a time sink
15:02 TD which APIs?
15:02 _dr did apple give a reason? or simple 'violation of tos, blah!'
15:02 HM just a mini bitcoin project
15:02 TD _dr: some kneejerk reaction by a lawyer there, iiuc. something like "money is hard! it's probably illegal somewhere in the world. simpler to just ban it"
15:02 _dr that's what i was expecting
15:02 TD HM: yeah it's a timesink but a very important one. otherwise you end up with mistakes like BTCGuilds
15:02 TD my goal is to make dangerous mistakes like that hard to do
15:03 HM got an example of a bad API you're dealing with right now?
15:04 TD bitcoinj :)
15:04 TD ok, most of it isn't terrible
15:04 HM yeah, but specifically
15:04 TD transactions have an associated object called TransactionConfidence which has data on things like how deep it is in the chain, how many peers announced it, whether it's in a side chain or not, etc
15:04 TD you can register a listener on that so you find out when it's changed.
15:05 TD the idea being, if a transaction reaches, say, 6 confirmations, and you want to do something at that point, you can just do that within your listener.
15:05 HM edge triggered or level triggered?
15:06 TD how do you mean? the listeners are run on any change. it's up to you to determine whether the transition is interesting.
15:06 HM right
15:06 HM so what's the problem with it?
15:06 TD ie, depth can go down as well as up
15:06 TD the problem is that there are unexpected re-entrancy constraints on what you can actually do inside that listener.
15:06 TD well, that's one problem
15:07 TD another problem is that the confidence data is sometimes modified in ways that users would not expect. for example, during a re-organize, confidence can change to intermediate states.
15:07 TD because of how the code is written.
15:07 HM sounds like a super-callback
15:07 HM trying to do too much
15:08 TD as a trivial example, it's not safe to actually re-spend the money from inside the listener, because the listeners run with the wallet locked. if you then try and lock a peer, you can trigger an inversion detection assert.
15:08 TD so it's just a mess.
15:08 TD i think the right fix is to make the whole confidence object immutable, and then batch up changes during certain operations so they don't trigger event listeners.
15:09 TD and then trigger the listener once at the end with the very latest state, without the wallet being locked.
15:09 HM why is it not immutable?
15:09 TD because confidence is inherently something that changes, right :) the only question is whether the whole object changes or whether parts of it are tweaked directly.
15:10 HM yeah, but from the callback it should be immutable
15:10 HM send the callback a copy and then hold no locks
15:10 TD yes. that's pretty much what i'm saying needs to change.
15:11 HM i guess it depends whether you need to enforce immutability over the entire callback
15:11 TD along with lots of other things
15:12 HM but there's a difference between notifications and pluggable functionality
15:12 TD the issue is not so much that the confidence object can change during a callback. it's more that the act of changing the confidence data is too tightly linked to the act of running the callbacks
15:13 HM this is what i meant by edge triggered, you need to decide whether the confidence is static while the callback is in flight.
15:13 TD so there are too many places where the confidence is mutated but it's not really safe for the wallet to be arbitrarily re-entered.
15:13 HM ah
15:13 HM spaghetti code
15:14 TD i wouldn't say it's spaghetti. but it's certainly complicated. it can be simplified a bit, but most of the complexity comes from the inherent nature of what some of these objects do. eg, processing a re-org is inherently a complicated operation
15:14 TD because you're essentially rewinding time and then replaying a new timeline, and what the API user really wants to find out, is the delta between those two timelines
15:14 TD and if there were any "merge conflicts" (i.e. double spends)
15:15 TD Satoshis code is already complicated and he didn't try to tackle this at all
15:15 HM sounds like you need a fullhog message queue and signal system
15:15 TD double spends trigger absolutely no useful form of feedback and there's not really any obvious place to fix that. if a tx is double spent away it just goes unconfirmed forever.
15:16 TD the callbacks are basically signals, of a form.
15:16 TD (you can add/remove listeners at any point)
15:16 HM but all callbacks should be called under a set of well defined preconditions
15:17 TD yeah. mostly they are. it's this one corner of the api where the callbacks are problematic.
15:17 TD the callbacks for things like peers being connected, disconnected, new blocks being processed etc are all ok
15:18 TD most of the classes allow for arbitrary re-entrancy.
15:19 TD i think we'll get this sorted. it's an ongoing effort. the api is ugly in other ways too, but i'm not aware of a better one.
15:19 TD at least it has good documentation and is mostly unsurprising
15:20 TD it also has example apps, things like that
15:20 grau TD: I solve the problem by disconnecting callbacks though a message bus. Events sent there have high abstraction e.g. complete reorg of blocks added/removed
15:21 TD well, if you register an event listener on the BlockChain then you get access to the list of blocks on both sides of the chain plus the split point.
15:21 TD the problem being that that isn't what is directly interesting to the users. they are thinking in terms of transactions (really: payments).
15:21 TD so they want to know that transaction XYZ just got double-spent away, or that it reached a certain amount of work done, etc
15:21 HM sounds like a new API is required
15:21 HM one that deals only in user concerns
15:21 TD and then on learning that they want to do things :)
15:22 TD HM: that's what the bitcoinj api is trying to do
15:22 TD HM: the devil is in the details, however. i'll fix it and then it'll behave exactly as users expect/need.
15:22 TD right now the API looks right, but there are odd restrictions on what you can do, and sometimes you get state changes that don't really map directly to user-interesting events. so it just needs tightening up.
15:24 grau HM: here it is
15:25 grau A new API, that has payment focus
15:26 TD yes, but it's currently under-specified. eg, if I use registerConfirmationListener you say it's called "if the transaction is confirmed up to depth 10 "
15:26 TD so what happens if it gets to 10 blocks, then a re-org moves it down to 8 blocks
15:26 TD and then it goes back to 10 blocks after a couple more blocks arrive
15:26 TD does it get called twice?
15:26 TD also, what happens if a re-org means that the tx will never confirm because its inputs were double spent? etc
15:26 TD so, it's just tough to get all these details right
15:26 HM hmm
15:27 TD grau: also that interface doesn't say anything about what thread it runs on or what re-entrancy is allowed
15:27 grau TD: yes it is called whenever confirmation number of tx changes
15:27 HM well you perhaps need to specify the maximum difference between blockchain heights before it's called
15:27 HM so like, depth of 10 with a minimum block branch distance of 4
15:27 HM or whatever
15:27 TD so what's the "up to 10" about?
15:27 lianj TD: maybe have a reorg callback in the callback and if its != null then this confirmation happend due to a reorg
15:28 grau TD: This is an API that runs by a message consumer detached from the kernel.
15:28 helo if you ask for 10, you'd be aware that the last few confirms could ~easily be undone, but that you can consider it permanent. so you'd not care about it reaching 10 a second time?
15:28 TD
15:28 TD this is the api i'm talking about
15:28 helo as for 10-deep reorgs... :/
15:29 TD grau: what does that mean? which thread does it run on? one i have to provide myself?
15:29 TD grau: also, do you have a wallet? most of the complexity in bitcoinj TransactionConfidence comes from interaction with the wallet.
15:29 TD if all you do is receive the data, it's not really so bad. it's when you want to start doing things based on that data like creating new spends and broadcasting them that the pain starts :)
15:30 xjrn td any netty headaches to speak of?
15:30 grau TD: this is not a single process. The thread that calls the API is coming from a thread pool that listens on a message bus. All requests to the server again go through the bus. Asynchronously.
15:30 TD netty 3.x has a really bizarre and confusing API too, which doesn't help. netty 4 looks a lot better but it's in beta
15:31 TD grau: ok so the listeners do have to be thread safe, yeah
15:31 grau Wallet for me is a constuctor of keys. not more.
15:32 grau Listeners are serial per topic.
15:33 TD that's not really how wallets work. they need to store transactions and create spends with them too
15:33 grau The real API is a subscription and publish protocol on a bus, the Java classes are just convinience wrapper
15:33 grau no they do not
15:33 grau this is just how yours nd satoshis work
15:33 TD how do you make a spend then?
15:34 grau You query UTXO to know what you can spend.
15:34 HM yeah
15:34 HM send a script filter over and have the service select the transactions
15:34 TD xjrn: i guess one minor annoyance is there doesn't seem to be any way to suspend the IO threads from processing things. not that it's a big deal. it'd be nice to have though.
15:35 HM so the main API deals in transactions and scripts and the wallet api abstracts over the top, only knowing about a few script forms?
15:35 xjrn TD see pm. i have that sussed.
15:35 grau HM: you send a list of adresses and server sends you UTX of them
15:35 HM why does the wallet API need to care about transactions?
15:36 HM inputs i mean
15:36 HM a 'wallet' is already a high level abstraction, it should be an idiot api
15:36 HM imho
15:37 TD if you assume access to a fully indexed UTXO set then yes, that is one way to do it. quite expensive to maintain though.
15:38 HM it's not necessarily expensive if you make your index data opaque and pass it over to the wallet subsystem
15:38 grau TD: yes, it is expensive, not an option for a mobile. But that is not my market either.
15:38 HM but this is just me philandering, API design *is* hard
15:39 grau HM: it is. The reason I have not yet announced beta of mine, because I still tune the API as I build on it.
15:41 Happzz i wonder why anyone thought placing the ip of whoever sends btcs in the blockchain
15:41 Happzz or is it not in the chain but just broadcasted?
15:41 HM it's actually a typical stateful server problem. indexes on the utxo set are server state and carry a hefty burden. keeping track of them transactions in your wallet system avoids that but breaks the abstraction
15:41 TD it's not even broadcasted
15:42 Happzz so why do people use tor to anonymize bitcoin?
15:42 HM Happzz: no IPs in the blockchain afaik?
15:43 Happzz HM i'm asking, not saying :o
15:43 TD connects to lots of nodes and records the IP it sees broadcast first.
15:43 HM IPs aren't future proof
15:43 TD sometimes that is actually not inaccurate
15:43 HM we might all move to meshnets in the future, that don't use IPs :P
15:47 xjrn HM: butterfly networks++
15:48 HM xjrn: never heard of them
15:50 ProfMac has anyone seen an IPv6 address in the data?
15:51 xjrn HM:
16:59 Belkaar If I use "encryptwallet" the client tells me that I need to replace my backups, because they will be made useless. How does this work? Private keys can not be invalidated or am I wrong?
17:01 Luke-Jr Belkaar: backups are only good for 100 transactions after they're made anyway
17:02 HM well BitSpend is a scary idea
17:02 Belkaar Luke-Jr: So I'm guessing the encryption-process deletes all unused keys in the pool?
17:03 Luke-Jr Belkaar: it marks them as used
17:03 Luke-Jr Belkaar: the assumption being it's best to avoid using potentially-compromised keys
17:04 Belkaar I see. The message in the Qt-Client suggests that it's no longer a Problem if an unecrypted wallet backup leaks, because it's useless now.
17:06 Belkaar But that would only be true if all coins are moved to a fresh address after encryption. Maybe there shoul be an option for that.
17:18 gmaxwell Belkaar: even _if_ you have no funds something must be done with the keypool keys, and doing that invalidates the backups.
17:18 Belkaar gmaxwell
17:19 Belkaar gmaxwell: I get that. I'm only concerned about the false sense of security it implies. "Encrypt your wallet and the unencrypted backups are useless (to thieves)"
17:23 Belkaar gmaxwell: And that is true for the unused addresses but not for the already used ones containing coins.
17:24 MobGod gavinandresen you therre
17:26 gavinandresen no, I'm here
17:28 jgarzik thar he blows, an ump like a snowhill
17:30 Luke-Jr MobGod: I'm there.
17:30 helo it does seem sensible that the client would offer to send all of your plain privkey coin to an encryption-protected address after encryption is enabled... but i'd bet that's been ruled out already for a good reason.
17:30 MobGod Luke-Jr where?
17:30 Luke-Jr MobGod: there!
17:30 MobGod great
17:31 MobGod trying to install name coin on my mac
17:31 MobGod would you be able to provide any help ?
17:31 Luke-Jr MobGod: #namecoin
17:31 MobGod i'm there
17:31 MobGod :)
17:31 Luke-Jr that's where to ask for help
17:32 MobGod i did with not one answer for 10 hours
17:32 MobGod i no gavinandresen runs a mac thats why i was going to ask him
17:33 Luke-Jr MobGod: probably because namecoin is dead
17:33 gavinandresen I don't know nuthin about namecoin
17:33 alexwaters cosbycoin is where it's at
17:34 MobGod Luke-Jr why do you feel it's dead
17:34 gavinandresen I think it is dead because it was an ill-conceived project from the start, but that discussion would be off-topic here
17:34 Luke-Jr MobGod: you just demonstrated why. and it's off-topic here in any case
17:35 Luke-Jr gavinandresen++
17:35 MobGod i guess that does make sense :P
17:48 midnightmagic namecoin is not quite dead yet.
17:48 Happzz it'll be soon
17:48 Happzz it's useless.
17:48 midnightmagic Happzz: nope.
17:48 jgarzik it's still merge-mined by a few big pools. who knows if it's actually used.
17:48 warren midnightmagic: if someone updates it to be modern and cleans up the half-outdated website, maybe.
17:49 warren also the "registars" are charging 1 BTC? that's a bit much
17:49 midnightmagic warren: There is no website.
17:49 warren jgarzik: I tried half of the listed domains last week, most are dead
17:49 warren midnightmagic: dotbit?
17:49 K1773R warren, if you do it yourself it costs like nothing
17:49 midnightmagic warren: That is not namecoin's website.
17:49 K1773R midnightmagic, dotbit IS namecoin's website
17:50 midnightmagic K1773R: Nope.
17:50 midnightmagic Vince is gone, there was no website, nor is there.
17:50 warren well, whatever the situation is, plenty of reasons why nobody is using it
17:51 warren btw, why is it that merge mining works with getwork but not stratum? limitation of the protocol or just implementation?
17:52 K1773R you get MM work with the RPC call getworkaux <aux> and since work generation is local in stratum, this wont work
17:52 K1773R stratum would have to be adjusted/changed to see it working
17:58 jgarzik stratum just does dumb concatenation, so merged mining seems possible?
17:59 K1773R yes, but not with the current implementation, it would have to be changed a bit on the server's side
17:59 K1773R well, a bit is a underestimation
18:03 jgarzik as long as stratum mining proxy uses GBT, should be straightforward :)
18:06 sipa hmm, how acceptable would it be to make bitcoin depend on GMP?
18:06 Luke-Jr warren: merged mining works just fine with stratum and GBT
18:07 warren Luke-Jr: oh? ok, crappy other implmenetations then.
18:07 Luke-Jr K1773R: which implementation? Eloipool does it fine
18:07 Luke-Jr sipa: ?
18:09 jgarzik sipa: ?
18:09 sipa Luke-Jr: i'm writing a secp256k1 implementations from scratch, and there are a few operations which still need a bignum library; either OpenSSL or GMP can provide those, but GMP is significantly faster
18:09 jgarzik Ah, I parsed that as GetMemoryPool
18:09 Luke-Jr ACTION also :p
18:09 sipa oh, i mean gnu multiprecision library
18:10 Luke-Jr sipa: we already have OpenSSL, and CBigNum?
18:10 sipa Luke-Jr: as i said, OpenSSL is slower
18:10 jgarzik sipa: do those operations occur outside of 256-bit and 160-bit integers?
18:10 jgarzik RE "there are a few operations which still need a bignum library"
18:11 sipa jgarzik: yes
18:11 sipa it's operations with scalar coefficients
18:11 sipa so not field elements (which are modulo 2^256-p)
18:11 gavinandresen I've finished my big shutdown cleanup spring cleaning:
18:11 gavinandresen ???still need to test on Windows before making it a pull request
18:13 sipa jgarzik, Luke-Jr: hardly an urgent issue - there are other options available like implementing the necessary operations directly so no dependency is needed
18:14 sipa Luke-Jr, Luke-Jr: just wondering whether there are legal/ideological/technological/religious/... oppositions against a GMP dependency
18:14 jgarzik sipa: picocoin.git is monitoring your efforts closely ;p
18:14 jgarzik sipa: surely it's LGPL'd, so works even with proprietary software
18:14 sipa it's LGPL indeed
18:14 Luke-Jr sipa: LICENSE="LGPL-3"
18:14 jgarzik sipa: most of these Younger Kids(tm) seem to not care about licenses, as long as the software works
18:14 jgarzik :)
18:14 Luke-Jr can't think of any problems
18:15 sipa to clarify: best performance on OpenSSL: 165us, best performance on GMP: 136us
18:15 sipa that is for a full ECDSA signature check, including pubkey decompression
18:15 sipa not including any bitcoin-specific processing like calculating the message hash
18:16 xjrn sipa seen ?
18:16 Luke-Jr GMP seems to have no dependencies of its own..
18:18 Luke-Jr seems KDE, GCC, and GHC depend on GMP, so it should be a standard part of most OS
18:18 Luke-Jr so probably the only real cost is in Windows binary size increase, which IMO doesn't matter much
18:18 sipa it's really just one operation for which it matters, modular inversion, and it's only used twice per signature check; but OpenSSL needs 26us for it, and GMP only 3us
18:19 Luke-Jr sipa: any reason not to submit a patch to OpenSSL to improve it there?
18:19 sipa Luke-Jr: that's also an option - i haven't investigated where the performance difference comes from, as both libraries use assembly-based optimized code at several levels
18:20 sipa but somehow i like the idea of not depending on OpenSSL...
18:20 fiesh there's also, offering a less restrictive license
18:20 Luke-Jr well, I imagine removing the OpenSSL dependency is more complicated than just using GMP
18:20 fiesh maybe worth checking out how fast it is
18:20 sipa Luke-Jr: if we stick to RPC-SSL support, ditching OpenSSL is no option at all
18:21 Luke-Jr fiesh: which isn't even packaged in some OS, let alone a standard component???
18:21 fiesh and then there's the original bsd library... what was it called again
18:21 warren Is this stuff for like 0.9?
18:21 fiesh Luke-Jr: of course I mean integrating it into the bitcoin source
18:21 _dr sipa: did you have a look at
18:21 Luke-Jr sipa: I'm not sure why we even support RPC-SSL considering exposing it to an untrusted network is unsupported/discourage
18:21 Luke-Jr fiesh: even worse
18:21 sipa warren: who knows; at this point, it's just experimenting
18:22 xjrn sipa mguanard and jfalcou on freenode maintain nt2 in #nt2 oddly enough. you'd be able to use headers-only afaik
18:22 Luke-Jr although, it looks like BSDMP is a drop-in alternative for GMP, so coding for GMP should work fine
18:22 warren sipa: I welcome this very much as Fedora/RHEL lacks ecdsa
18:22 _dr they do several karatrusba runs to approximate the 256bit mult with carryless 64bit mult
18:22 warren sipa: in openssl
18:22 warren hmm, what GNU library is this?
18:23 Luke-Jr nevermind, BSDMP makes no sense
18:23 fiesh oh, the traditional bsd library is just called libmp
18:23 sipa _dr: seems interesting; there are certainly multiple improvements possible still with the field operations
18:23 fiesh Luke-Jr: yeah, seems to be not well enough developed, I agree -- I found it accidentally when looking for the original BSD library
18:23 sipa warren: i don't feel like being responsible for the entire ECDSA verification stack in bitcoin, so i'm not eager to have this merged without very serious review anyway
18:24 warren sipa: you're implementing one embedded in bitcoin itself, or switching to another library?
18:24 fiesh oh but the current bsd mp implementation just uses openssl, hehe... so no gains there
18:24 Luke-Jr lol
18:24 sipa warren: at this point, it's even independent of bitcoin; i just started writing an ECDSA implementation specific for secp256k1 from scratch, with all optimizations i know of
18:25 sipa warren: and given that it's significantly faster than OpenSSL at this point, it would be nice to take advantage of it
18:25 _dr sipa: then have a look at that paper
18:25 _dr my guess is you won't get faster than the guys at intel, their crytpo library is the fastest since they know their hardware
18:26 sipa well, this is for one specific curve
18:26 xjrn looks like a static eval field-day
18:26 _dr they do a clever trick. while they only use a 64bit multiplication, they use the ymm registers to keep the results of the intermedia steps in the 256 bit vector registers
18:28 sipa _dr: i don't want to go into maintaining our own assembly-optimized versions
18:28 _dr also, the carryless instruction is part of aes-ni, which is availabe in all core cpus so you won't have to use a sandy bridge cpu
18:29 _dr sipa: you can always use intrinsics, but IMO maintaining a well written and documented assembly kernel is doable. you'd only have to write the 'hot' parts in actual assembly
18:30 sipa sure, and those are very limited
18:30 sipa my current field implementation uses 5 64-bit integers, each holding 52 bits
18:30 sipa so no carry is needed to add the result of several multiplications
18:31 _dr i think they list their code at the end of the paper, they claim a 6x speedup comparing to openssl 1.0.0a wrt 256 ecdsa signing
18:31 _dr no sorry, 6x speedup wrt verification! signing is slower, but who cares ;)
18:31 sipa i'm already close to a 5x speedup over OpenSSL :p
18:32 _dr wow, very nice
18:32 sipa but there are secp256k1-specific optimizations at every level
18:32 sipa so that 5x speedup is certainly not (only) because of fast field operations
18:33 sipa 5x is exaggerated i think; it's closer to 4x
18:34 _dr and once again we see that single-core optimization is worthwhile
18:34 xjrn sipa straight c?
18:35 K1773R Luke-Jr, didnt know that, ty! gonna check it out :) is it possible to mine on multiple chains or just 1 altchain?
18:36 _dr and we'll get another boost from HNI once skywell is out. i think they'll actually do 128bit ints
18:37 sipa hni?
18:37 Luke-Jr K1773R: Eloipool depends on a fork of merged-mining-proxy to handle that all; it should handle unlimited chains so long as you have Eloipool's available to it
18:37 Luke-Jr _dr: GCC already has 128-bit ints on some platforms
18:38 sipa Luke-Jr: GCC does; hardware doesn't
18:38 _dr Luke-Jr: I'm talking about native
18:38 Luke-Jr sipa: AFAIK, GCC only provides it when the hardware does - including modern x86
18:38 sipa Luke-Jr: no
18:38 _dr sipa: AVX2. the documents says The 128-bit numeric processing instructions in AVX cover floating-point and
18:38 _dr integer data processing across 128-bit vector and scalar processing
18:38 sipa Luke-Jr: the hardware has a 64bit * 64bit multiplication with 2x 64-bit output
18:39 sipa Luke-Jr: GCC's __int128 is an aggregate of two 64-bit variables that is needed to capture the output of such an instruction
18:39 _dr Luke-Jr: i can assure you there's no support for 128bit mult in hardware yet (unfortunately)
18:39 sipa Luke-Jr: but adding two __int128 together for example will be translated to separate additions on separate registers
18:39 _dr at least for intel cpus. there may be more exotic architecures that provite it, though
18:40 Luke-Jr :<
18:40 sipa so in a sense there is an extremely-limited support for 128-bit types in the hardware indeed: namely as the result of a 64-bit * 64-bit multiplication
18:42 gmaxwell It just carries on the same thing i386 had-> a 32x32->64h/l which was also inaccessible in C except via long long... amd64 just scaled up the registers so you got a 64x64->64h/l
18:44 _dr i really wonder why they don't have them though. can't be that hard, can it? avx has 256 bits, but no integer instructions at all
18:45 Luke-Jr _dr: AVX2 is supposed to have 256-bit int as well I hear
18:46 gmaxwell _dr: the avx2 stuff adds all the integer operations, but those are vector registers. The complexity of doing 8 32 bit multiplies is much lower than 2 128 bit ones. :P
18:47 _dr yeah I know, but... if you have the large registers, why not offer an instrction that does the 256 bit multiplication in hardware, but slower than a 32/64 bit multiplication
18:47 _dr all these instructions are pipelined, so using a larger operad size would just increase the latency
18:54 _dr sipa: is your code on github?
19:03 HM faster verification is awesome
19:03 HM i wonder how well it'd run on arm
19:07 ThomasV can ecdsa be used for message encryption?
19:08 Luke-Jr no
19:08 Luke-Jr but it could sign a key that is
19:08 ThomasV oh:
19:08 gmaxwell the ECC keys we use certantly can be, but thats not ECDSA
19:09 HM public key encryption is a matter of some complexity, especially if you want authentication using the same key
19:09 ThomasV gmaxwell: yeah, I meant the keys
19:10 warren sipa: sorry disappeared earlier, power went out
19:11 warren sipa: what's the link again?
19:34 sipa warren, _dr:
20:11 _dr thanks
20:18 egecko ok.. quick query.. what are command line flags to have it read from block chain from one directory and the wallet from a different one? i see -datadir but it looks like block chain and wallet have to be in the same directory, is that correct?
20:18 weex yup
20:18 egecko bummah!
20:18 egecko oh well
20:23 sipa_ egecko: symlinks to the rescue
20:24 rdponticelli1 egecko: Just use a symlink
20:25 egecko yeah, that'll get done once this node finishes reindexing the block chain
20:26 egecko anyway to tell when a daemon is done reindexing?
20:26 egecko other than watching for the CPU usage to drop?
20:43 egecko well at least bitdozer still works with 0.8!
20:46 nick___ hello
20:46 sipa what is bitdozer?
20:46 nick___ no idea
20:47 egecko its a windows client for access bitcoin servers
20:47 egecko windows phone
20:48 egecko pretty useful if you wanna send coins from your windows phone :)
20:48 nick___ anyone have success with the socketio / websockets c# projects on github? i load the examples and it connects but no trade messages are ever sent to my client by the server
20:48 alexwaters ACTION envisions a bulldozer paid for in bitcoin - bulldozing a local bank - because of bitcoin
20:49 nick___ with socketio i never get back confirmation messages that my subscriptions were ack, and i've tried dozens of different types of message formats to try to get a subscription to work
21:52 HM hmm the exchange rate for selling my $ers seems favourable right now
21:53 HM (in to ?? as well as Bitcoin)
21:53 HM err not Bitcoin, just ?? :P