Transcript for #bitcoin-dev 2017/09/15

00:09 lrvick looks like I was confusing rpc calls from 0.10 docs. it is happy now
00:11 CryptAxe yay!
07:44 Cocodude Hey all. Just upgrading to v0.15 (good work btw), but there's one change you've put in that I don't like and I think it's not far off being a bug.
07:45 Cocodude You now require that the wallet.dat file is a regular file, and don't allow it to be a symlink (Error: Error loading wallet wallet.dat. -wallet filename must be a regular file.)
07:45 Cocodude And if I specify the real wallet in the -wallet command line, the client complains with "-wallet parameter must only specify a filename (not a path)"
07:46 Cocodude This is a bit of a pain - I store the wallet on an encrypted partition so this now stops me from being able to do this effectively. Why was this change put in (especially the lack of love to symlinks)?
07:50 esotericnonsense i believe that it's to enforce locking semantics on the wallet file (it has always been necessary that the wallet file exist in the datadir and symlinking is a hack around that)
07:50 esotericnonsense e.g. two bitcoin instances operating on the same wallet file would be bad news
07:51 Cocodude Ah, yes, that would be bad
07:52 Cocodude I wish there was a way to override this check then I guess, but that sounds like a lot of work which only benefits people doing things in a special way, like me
07:53 Cocodude I don't imagine I'm the only one doing this though - I don't want to keep the entire blockchain encrypted (it's not exactly a secret) but I want the wallet to be. It provides some security as everything important as I have encryption at rest.
07:53 Cocodude I can't think of a neat way to do this any other way off the top of my head
07:53 arubi how about having the datadir on the encrypted filesystem and symlink the big directories out?
07:54 esotericnonsense i'm not sure why you wouldn't keep the blockchain encrypted, overhead is ~zero on modern CPUs
07:54 Cocodude "neat way" :)
07:54 esotericnonsense aesni is 3GB/sec per core
07:54 Cocodude esotericnonsense: Encrypted for me is also on a SSD as the wallet has a lot of I/O on it
07:54 Cocodude And SSD space is limited
07:55 esotericnonsense you could do it backwards and symlink blocks/chainstate ? :P
07:55 arubi that's what I was suggesting :)
07:55 Cocodude Yeah, it's possible, but even more ugly than just symlinking the wallet
07:55 esotericnonsense arubi: sorry, didn't see that :P
07:56 Cocodude arubi/esotericnonsense: It would work, I agree, but what happens when Bitcoin Core adds another special folder/file there that I don't really want on the SSD?
07:56 arubi well like you said, it's a specialized setup. you'll have to adapt
07:56 Cocodude Thinking about it again, do you think it really is that specialised?
07:57 esotericnonsense i'm confused by 'wallet has a lot of IO on it' really, unless you're using it as a merchant and chucking many tx per second at it
07:57 Cocodude Wallets do have a totally different security profile to the rest of the directory
07:57 Cocodude esotericnonsense: Guess what, I am!
07:57 esotericnonsense :P
07:57 arubi it does have a bunch of io
07:57 Cocodude OK, not many per second, but often a few a minute
07:57 Cocodude When a new block comes in, there's lots of fun that happens with wallet IO too
07:57 esotericnonsense my understanding might be outdated but I believe that work done on wallet.dat can leak outside of the file anyway
07:58 esotericnonsense only private keys are treated with care
07:58 Cocodude That's the only security-critical thing though. If metadata or whatever spills out, then I can live with that (although I thought all that was in the wallet too)
07:58 arubi it was, but only keys are encrypted
07:58 esotericnonsense can you not use the built in encryption?
07:58 arubi it is*
07:59 Cocodude It's possible, but it's a lot neater all round to just have block encryption on a partition for me
07:59 arubi wait, so it's unencrypted by itself?
07:59 Cocodude Yes
07:59 arubi and core is running, and it is loaded?
08:00 Cocodude Yes
08:00 Cocodude Now you're going to tell me that's a stupid thing to do... :/
08:00 arubi not sure how encrypted filesystem helps?
08:00 esotericnonsense Yes
08:00 Cocodude Ah, phew :)
08:00 esotericnonsense ;)
08:00 Cocodude It helps as I have encryption at rest
08:00 Cocodude Someone steals the server, I should be safe
08:00 arubi yea, but the wallet is wide open all the time
08:00 Cocodude Not if they've stolen the server and it's powered off or rebooted!
08:00 esotericnonsense do you have encrypted swap? (now i'm just being an arse :P)
08:01 Cocodude Or just taken out the HDDs or whatever
08:01 Cocodude esotericnonsense: Yes, I do
08:01 esotericnonsense :D
08:01 arubi yea if the pc turns off then it's encrypted, but most times it's not off
08:01 esotericnonsense it might be better to ask this in #bitcoin-core-dev. the specific commit seems to be here: https://github.com/bitcoin/bitcoin/commit/a6da027d83e48f05c933149ff89c9b9ad5ced915#diff-b2bb17478...
08:02 arubi Cocodude, is that actually a hot wallet then?
08:02 Cocodude arubi: Yes
08:02 Cocodude esotericnonsense: Thanks for pointing out the commit
08:02 arubi I mean, hopefully larger funds are stored fully encrypted and offline right?
08:02 Cocodude Yes, indeed
08:02 esotericnonsense i suppose the easiest option if you're fully committed to doing this
08:03 Cocodude I'm only on about the hot wallet
08:03 esotericnonsense is to rip that out and recompile
08:03 arubi alright, understood
08:03 esotericnonsense but i'd try and work out if there's a good reason for it
08:03 arubi yea I guess that's the best you can do in this case if the wallet is used for sending a lot
08:03 Cocodude esotericnonsense: Yeah, it would be trivial to just comment out a few lines, but it's only a very short term solution
08:04 Cocodude arubi: Indeed. There can't be negligable funds in the hot wallet either as the amount needed in there fluctuates quite a bit).
08:04 esotericnonsense if you're using rpc you can also just use the built in encryption and unlock it with walletpassphrase for an absurd amount of time, i'm not sure if there's an upper bound
08:04 Cocodude esotericnonsense: I need a new wallet as well for that
08:04 Cocodude My wallet is, ehm, quite big in terms of size
08:04 arubi I think multiwallet is in 0.15.0 ?
08:05 Cocodude Yeah I think that would be helpful in migrating.
08:05 esotericnonsense it should be doable, it'll just take a while, no? i'd be surprised if the enc/dec wasn't using aesni
08:05 Cocodude There is some slowdown with my large wallet (takes a few seconds to send a tx) so some code core which is O(n) or similar
08:06 Cocodude esotericnonsense: Doesn't stop the requirement to have the wallet on a SSD - the wallet really does appear to use a lot of I/O.
08:06 esotericnonsense ah yeah, that's true
08:06 esotericnonsense well, wrt 'another special file/folder you don't really want on the SSD' and the symlinking blocks/chainstate approach
08:07 esotericnonsense it seems like it would be obvious ahead of time if another large set of data is produced
08:07 esotericnonsense e.g. chucking a few extra megs on the SSD doesn't seem to be that much of a problem, only if it's actually say, >1GB
08:08 Cocodude That's true
08:08 Cocodude It just feels a lot more messy. There's only one thing I want moved, so it seems nicer to move that one thing (rather than everything *but* that one thing)
08:09 arubi well having peers.dat , mempool.dat, debug.log .. etc on the encrypted ssd will help with io and privacy if the pc is stolen
08:09 esotericnonsense the entire contents of my .bitcoin folder aside from chainstate and blocks is about 15MB, you might have a bigger debug.log and mempool.dat could be a few meg i suppose
08:09 esotericnonsense yes, that too
08:09 esotericnonsense debug.log especially
08:10 esotericnonsense having chainstate on SSD would probably improve performance too if you can spare the 3GB
08:11 esotericnonsense depends on your dbcache setting
08:11 Cocodude Yeah, just the chainstate, blocks and debug.log really shouldn't be on the SSD
08:11 Cocodude Hmm, good point about chainstate
08:11 Cocodude Actually, I just use the default dbcache values. With the wallet on the SSD, I think I'm more CPU bound because the wallet is so huge.
08:15 Cocodude Just thinking about this, all this commit does is to stop the wallet from being specified twice on the -wallet command line (one regular, one symlink to that file). I think that sounds like a more niche "security improvement" than my setup.
08:15 Cocodude It wouldn't stop, for instance, two instances of Core running, both pointing to the same wallet
08:16 esotericnonsense Cocodude: you can't run two copies of bitcoin from the same datadir without hacking
08:16 Cocodude Oh yes, the -wallet has to be in the same datadir as you can't specify a path. Got it.
08:16 esotericnonsense yes :P
08:22 Cocodude I've added my comment to https://github.com/bitcoin/bitcoin/pull/10885 so we'll see if it gets a response
08:22 Cocodude This is going to be especially fun as other alts pick up on this and I need to shuffle everything around for Litecoin, Doge etc.
08:26 esotericnonsense thinking about it, if that's the rationale (two wallet files symlinked to each other in the same instance), then it should be possible to determine that at startup, if a bit of a faff
08:27 esotericnonsense it seems like there must be a way to determine if a file and a symlink point to the same file, if not, you could hash them both and compare (though i suppose with a large wallet that might take a while :P)
08:29 Cocodude Youch, that seems like a lot of work
08:29 Cocodude Honestly, I think what's been done here is too much handholding anyway (at the expensive of people like me).
08:29 Cocodude If you're symlinking, you're doing something special anyway and should know what you're doing
08:30 Cocodude Also, hashing wouldn't work - it would be reasonable to have the same, precise wallet on there twice
08:30 Cocodude (just two physically different files, but the same contents)
08:32 Cocodude Got a nice response at https://github.com/bitcoin/bitcoin/pull/10885
08:33 esotericnonsense yeah, that's about in line with my expectation, that wallet.dat is not self contained
08:34 Cocodude I'm a little confused though - what does it rely on that, say, a -rescan can't solve?
08:34 esotericnonsense it's been a while since I looked at it
08:38 esotericnonsense another 'supported' way unless you have very very limited storage would be to move everything and prune to the minimum, then you'd have about 4GB on SSD (plus wallet.dat) in total, though that breaks rescan
08:38 esotericnonsense it's a tricky one
08:38 Cocodude Yeah, I don't want to do that really, and like running a full node for altruistic reasons
08:50 Cocodude Now sipa is saying, "There are no dependencies between the wallet and the rest of the datadir, in either direction."
08:50 Cocodude That's what I thought, but didn't sipa just say the opposite before?
08:51 esotericnonsense wallet, not wallet.dat
08:52 esotericnonsense bitcoin wiki is pretty out of date, but seems to claim that there's a database/ folder that stores bdb state
08:53 esotericnonsense my nodes are running with disablewallet so don't have it here
08:53 Cocodude Yikes, does that mean I should be backing that up too?
08:54 esotericnonsense it only exists during runtime, so i expect that if there's a hard shutdown, that it'll be used for recovery
08:54 Cocodude Indeed, which means it's probably best to store it in the same partition as the wallet.dat
08:56 arubi that file has data like BDB2565 [4][397132] Non-transactional update, log type: 2, fileid: 1.
08:56 arubi BDB2549 [4][397219] Checkpoint record, ckp_lsn: [4][397088], timestamp: Wed Sep 13 20:51:20 2017
08:56 arubi . Total checkpoint: 4
08:56 esotericnonsense by the way, it's great to see you guys are still running, feels like an eternity since I first saw you pop up :)
08:57 arubi (yes looks like it's related to bdb and wallet.dat)
08:57 Cocodude arubi: Sounds like it'll be quite I/O intensive too
08:57 arubi that's right
08:57 arubi I mean, I'm doing a couple of addnewaddress now and it's writing a lot of that ^^ each time
08:58 Cocodude Is this the database folder?
08:58 arubi yea, the 000000..log file
08:59 arubi run `db_log_verify -c | less` (if you have db-utils)
08:59 Cocodude Yeah. I guess a flush is when they's all put into the actual wallet.dat?
08:59 Cocodude Nah, I just have the library here
08:59 wumpus yes, the database/ folder has bdb state, it's compacted into wallet.dat extremely regularly so it will usually be empty
08:59 Cocodude Cheers wumpus
08:59 Cocodude I guess then, technically, it's a little safer to store that in the same place I'm storing the wallet.dat
09:00 wumpus after a clean shutdown, there is no need to backup anything but wallet.dat
09:00 Cocodude As if I want to recover from a hard crash, they're more likely to be in sync on the same parittion
09:00 wumpus yes
09:00 Cocodude Yeah, only talking about hard crash, and I'd probably consider using a backup anyway
09:01 Cocodude wumpus: What's your take on my rant?
09:01 wumpus what I usually do is place the "blocks" folder on another partition, sometimes "chainstate" too, and keep the rest together
09:01 Cocodude Ooooh, so I'm not the only one doing crazy symlink stuff
09:01 wumpus no
09:02 Cocodude wumpus: Do you think the symlink check is a good thing for wallet.dat or a bad thing?
09:03 wumpus by far not, I do crazy hardlink stuff too, like https://gist.github.com/laanwj/3c4614a23e072cbb3d39090da1834a68
09:03 wumpus what should it do if the wallet is a symlink?
09:03 Cocodude It won't start, quite simply. There's a new check in there.
09:03 wumpus probably just reject it, as all database state is expected to be in one directory by bdb
09:04 wumpus oh that's good
09:04 Cocodude Oh, I don't think it's good
09:04 Cocodude If I symlinked both the database folder and wallet to a different partition, it would still error out
09:04 wumpus bdb works with a database *directory*, where all databases need to be
09:05 esotericnonsense so you'd basically want wallets/database and wallets/wallet.dat
09:05 Cocodude Yes, just clicked
09:05 esotericnonsense within the datadir or otherwise, and then this wallets directory could be symlinked off
09:05 wumpus allowing symlinking would allow crazy things like linking multiple wallets from multiple partitions w/ multiwallet
09:05 wumpus don't symlink bdb files
09:05 wumpus could add a walletdir option that sets a different wallet database directory for the data directory, if you really need that
09:06 wumpus s/for/from
09:06 Cocodude Yes, that's what sipa is suggesting
09:06 wumpus but all bdb files need to be together and not be symlinks
09:07 wumpus ok, that's the way forward then
09:07 Cocodude Sounds like the neatest way, but not a quick solution.
09:17 wumpus doesn't sound particularly difficult or much work, but maybe I'm missing some subtlety
09:18 wumpus the other option would be to symlink the *non-wallet* things as I do
09:18 Cocodude Yeah, agreed it's an option, but a bit yucky.
09:19 wumpus symlinking the wallet is yucky and brittle too
09:19 Cocodude Agreed it doesn't sound too difficult really, but anything code-wise is likely to take some time to go into a stable release
09:19 Cocodude It's a bit yucky, but it makes mores sense in terms of "I want this file secured, so I'll move it" rather than "I want this file secured, so I'll move everything but this file"
09:20 wumpus that's all good in theory, but in practice bdb works with a database context directory, not a file, so it's really a directory that you need to secure
09:21 wumpus e.g. your scheme falls apart if you have multiple wallets
09:21 Cocodude Yes, understood there. We're both agreeing that the wallet really should be its own directory.
09:21 wumpus yes
09:35 Cocodude Is there an enhancements tracker for Core?
10:40 wumpus github issues?
10:40 wumpus or the release notes, for a summary per release
10:43 Cocodude No, I mean to potentially raise the wallet directory as a new enhancement.
10:43 Cocodude Not sure if github issues is the right place as the default contents talks about everything to do with bugs
10:59 wumpus well you can certainly create an issue for an feature request, though the chance of someone picking it up is slim, unless another developer happens to have exactly the same need
12:01 Shaun3811 Hello
12:12 Shaun3811 chatty bunch then
14:53 cedenday Does anyone have more info on how SegWit is going to be supported in QT?
14:55 cedenday IMO they should be default with some sort of flag disabling that behaviour.
15:00 wumpus I don't think that's decided yet, there is nothing implementing that at least
15:01 wumpus probably will be some setting
18:26 esotericnonsense Cocodude: see https://github.com/bitcoin/bitcoin/pull/11343
18:27 esotericnonsense Cocodude: confirmation there that the database folder can store sensitive information if the wallet is unencrypted