Transcript for #bitcoin-dev 2017/09/29

10:56 lxer ZMQ zmqpubrawtx seems to send 3 messages for each transaction, the second having the actual data, so what are the other 2 for?
18:53 esotericnonsense lxer: what do they look like? :)
19:21 lxer esotericnonsense: ok, got it. first one translates to 'rawtx' . the 3rd one is a hex value like 9a510500 . what is it for?
19:26 esotericnonsense lxer: so the first one is the published zeromq topic (in multipart messages it's always that, that's how zmq pubsub works)
19:28 esotericnonsense the last one is a sequence number (little endian according to the code). looks quite high, not sure if it starts from 0, how long has bitcoind been running? :)
19:31 lxer are you sure that is how pubsub works? I already set a topic filter in socket.setsockopt(zmq.SUBSCRIBE, "rawtx") , shouldnt that filter it out already? I've used ZMQ for other projects, but never the pubsub, so perhaps that is different from what I'm used to.. ?
19:31 esotericnonsense lxer: yes i've worked with pubsub rather a lot in the past :)
19:32 esotericnonsense lxer: it's not echoing your subscription back to you, rather it gives you the topic published
19:32 lxer I usually get just 1 message.
19:32 esotericnonsense for example if the publisher sends out on "CHOCOLATEBAR" and you subscribe to "CHOC" the messages will be ["CHOCOLATEBAR", *parts]
19:36 cluelessperson how large of a database do you think a bitcoin postgresql database would be?
19:36 esotericnonsense cluelessperson: it really depends what you're storing
19:37 cluelessperson esotericnonsense: I'm thinking tables for, blocks, transactions, addresses
19:38 cluelessperson esotericnonsense: I'm thinking about creating large flow visualizations
19:39 lxer there have bee some projects about that. even making nice 3D images of transactions
19:39 esotericnonsense cluelessperson: what i mean is the format of the things that you store. for example if you're indexing by txid anyway you can drop signature data.
19:40 esotericnonsense if you store some big verbose transaction in a nonraw format then it will be much larger, etc.
19:40 cluelessperson esotericnonsense: sorry, why can I drop signature data?
19:40 esotericnonsense cluelessperson: well because you don't care for that specific purpose right.
19:41 cluelessperson esotericnonsense: I want to be able to more easily/quickly parse through all the trnasactions
19:41 cluelessperson esotericnonsense: render a data visualization
19:41 cluelessperson but we're talking like 300million
19:42 lxer esotericnonsense: but then why is it sending the zmq topic again as a message? I never needed that when using some request/reply pattern , and the 'topic' still works as usual.
19:44 esotericnonsense lxer: it's not 'again'
19:44 esotericnonsense it's just how pubsub works
19:46 esotericnonsense often you'll encode data in it because the topic needs to do that anyway
19:48 esotericnonsense for example if you have some stock ticker app or whatever and you have TICKER:AAPL, TICKER:GOOG, etc, if someone subscribes to TICKER, they get everything and can distinguish based on the topic rather than having to also include that in the payload
19:49 esotericnonsense i'm not that up on the internal workings of ZeroMQ (it's been a while since I went digging there) but I suspect it can't even work through proxies if the topic weren't sent
19:49 esotericnonsense (because the proxy wouldn't know where to route the messages)
19:50 lxer I remember it differently. the topic filter is already set when creating the socket: socket.setsockopt(zmq.SUBSCRIBE, topicfilter) , and the messages from then on are already filtered by topic
19:51 esotericnonsense the filters can be set or unset at any time
19:51 lxer that is not the point
19:51 esotericnonsense say you have a bunch of publishers, with a hub in the middle, then some subscribers
19:52 esotericnonsense the subscribers will sub to the hub, the hub will subscribe to the publishers, and when the hub receives a message, it uses the topic to route to the subscribers
19:52 esotericnonsense without having that topic, it doesn't work
19:52 lxer a 'hub'? I always connect directly
19:52 esotericnonsense (I can see how in the simple pub -> sub model with no middleman it doesn't seem necessary)
19:52 esotericnonsense yeah, but zmq is designed to scale in that way
19:53 lxer no sorry, I think this design is not right.
19:53 esotericnonsense the point is that you could have many publishers on different boxes whilst a subscriber doesn't have to do service discovery, it just needs the hub
19:54 lxer there is no need to create so hub to have a second layer of filtering
19:54 esotericnonsense it's not for filtering purposes
19:54 esotericnonsense imagine your publishers are doing computationally intensive tasks
19:55 lxer no, if you want to scale through proxies, then simply make a new pub/sub on the proxies, with a topic filter
19:55 esotericnonsense you can't put them all on one box, so you route everything through a hub which just serves as a zmq proxy (this is actually a builtin, http://api.zeromq.org/3-2:zmq-proxy)
19:56 esotericnonsense imagine another case without proxy if you will
19:56 esotericnonsense (in fact i'm being silly for not having mentioned this earlier, sorry heh)
19:57 esotericnonsense if you set multiple subscription topics on the same socket, how would you distinguish them on receipt without having the topic?
19:57 esotericnonsense you'd have to encode that in the payload somehow
19:57 esotericnonsense but the topic is already there, so just use that :)
19:59 esotericnonsense perhaps my defence of pubsub mechanics is not to your liking, but, it is the way it is :P
20:00 esotericnonsense there are other weirdnesses in pubsub, like on the publisher end if you use verbose sockets you can catch the subscription messages and not bother to send if no-one is subscribed
20:03 lxer I'm using a pub/sub for some other project, where I just receive messages about a topic, send by the publisher (who set the topic also). always just 1 message, and not 3. and the topic-filter works as expected
20:03 cluelessperson esotericnonsense: what size are we really talknb about here?
20:04 esotericnonsense lxer: well you can send messages as topics if you want too
20:05 lxer no, that is not what I mean.
20:05 esotericnonsense i guess i'm calling it topic because i've never used it in that way, then the topic is just the payload
20:06 esotericnonsense i think you can just publish without an envelope and make the first part of the message what you subscribe to (or subscribe to "")
20:06 lxer I mean that it would just work the same if you left out message 1 and 3 , and only send the actual content (2nd message)
20:07 esotericnonsense lxer: i think i'd understand this better with a code example
20:08 esotericnonsense in python you could send non-multipart "something whatever" as a string, and subscribe to "something", it's the same just without the framing of multipart messages
20:08 lxer ok, it is friday night over here and I want to do something else. lets talk about it again later
20:08 esotericnonsense if you just send a blob of data with no topic then the subscriber can't filter messages
20:09 esotericnonsense it would work in the specific case of bitcoind if you only published txids or only block hashes, but if both were enabled then you wouldn't know which was which
20:09 lxer yes, the subscriber does filter, that is handled on another level, afaik.
20:10 esotericnonsense argh.
20:10 esotericnonsense the way you publish a message is to send a blob of data.
20:10 lxer + a topic
20:10 esotericnonsense you can send without a topic
20:11 esotericnonsense (it's just that if you do that there is no topic! :P)
20:12 lxer right, but the publisher (bitcoind) sends with topic 'rawtx'
20:12 esotericnonsense yes, and it sends in the multipart manner
20:12 esotericnonsense it could also send in one part, as "rawtx hash" instead of ["rawtx", "hash", "sequence"]
20:13 esotericnonsense err, "rawtx hash sequence"
20:13 esotericnonsense the sequence number is there so that you can detect dropped messages btw
20:13 lxer then, if I create a subscriber socket.setsockopt(zmq.SUBSCRIBE, "rawtx") , I only get 'rawtx' messages. So, no need to send a message with 'rawtx' to announce it.
20:13 esotericnonsense ????
20:14 esotericnonsense the publisher has to announce the topic it's sending on, that is what sending on a topic is?
20:14 esotericnonsense the first part of a multipart message, or the entire message if not multipart, is the topic
20:16 lxer 'topic' and 'message' are different things
20:16 lxer anyway.. i'm wanna get drunk and party :P
20:17 lxer so, if I have time I'll make some example code. is python ok?
20:17 esotericnonsense i have an example here, just a sec
20:17 esotericnonsense it is fine, yes
20:19 esotericnonsense eh, can't get it to work in asyncio with one process.
20:20 esotericnonsense basically, in python, make a subscriber to "TOPIC TOPIC". make a publisher that just does sock.send("TOPIC TOPIC PAYLOAD")
20:20 esotericnonsense that's the non multipart way
20:21 esotericnonsense alternatively you sock.send(["TOPIC TOPIC", "PAYLOAD"]), the client will get it by x = recv_multipart, x[0] will be the topic, x[1] the payload
20:21 esotericnonsense err sock.send_multipart*
20:21 esotericnonsense in both cases you're sending the topic, not sure what you're getting at :/
20:32 lxer ok.. I always used recv() and not recv_multipart . I think I get what you were saying now.
20:33 lxer anyway.. beertiem.
20:33 esotericnonsense got it working. https://0bin.net/paste/eUGjYXpOBpK05qao#qnrNNT3fXZS6KG5eu4dt85xvi3FutMXHqQqbnI531AK
20:33 esotericnonsense have fun :)
20:35 esotericnonsense right. i see now. after playing with it.
20:36 esotericnonsense if you send_multipart(['topic', 'payload']) and then recv, you will get two individual receive events of 'topic' and 'payload'.
20:36 esotericnonsense now I will get food. :P
21:53 lime_ Hey all, trying to compile a node on armv7l hf, getting an error saying G++-arm-linux-gnueabihf isn't installed
21:54 lime_ is it looking for a cross compiler?