How to host a Zcash litewalletd server via docker

In a prior post I documented how I used a docker-compose.yml file to make it easy to create and maintain a zcashd server. In this post, I add a litewalletd docker container that allows me to be entirely self-hosted in my use of lite Zcash wallets.

The litewalletd docker image has no documentation at all, but there is documentation for it over at Lightwalletd Instance Setup Guide. That gives a good starter to know port numbers, the path to mount a volume, and some switches.

My goal was to add a litewalletd service to the same docker-compose.yml file that hosted my zcashd container as I described in my a prior post. This would keep maintenance low, and even give me a chance to improve security on the zcash node, as I’ll describe.

System Requirements

The Zcashd node itself, when configured to support litewalletd, takes 362GB of storage as of this writing. Litewalletd itself adds another 13GB. Given an ever-growing blockchain, I suggest you only set a litewalletd server up on a machine on which you can dedicate at least 500GB of storage to it.

Zcashd alone, without the special txindex configuration required to support litewalletd described later in this post, takes under 200GB.

Preparing zcashd for a litewalletd client

litewalletd demands special abilities of a zcashd node that require opt-in. If the following lines are not already in your zcashd node’s zcash.conf file, you must add them.

server=1
txindex=1
insightexplorer=1
experimentalfeatures=1

After adding these lines to an existing zcashd node, you must tell the node to reindex the chain. Be sure to first stop your ordinary zcashd docker container with docker-compose down within the directory containing your docker-compose.yml file. Then run this command (or similar):

docker run --rm -v zcash_data:/srv/zcashd/.zcash -v zcash_params:/srv/zcashd/.zcash-params -p 8232:8232 -p 8233:8233 electriccoinco/zcashd --reindex 

This will start a temporary zcashd container that is dedicated to reindexing the chain. Note the --reindex switch at the end.

This took a week on my machine, and more than doubled the local storage space required. A few hours into reindexing, I accidentally hit Ctrl+C on this terminal, which stopped the container. I found that simply re-running the above command without the --reindex switch caused the node to resume the reindex operation rather than restart it.

Details of my experience are in the Troubleshooting section at the bottom of this post.

Adding your litewalletd service

With zcashd ready for a litewalletd client, we now add the litewalletd service and volume to the existing docker-compose.yml:

version: '3.1'
services:
  zcashd:
    image: electriccoinco/zcashd
    restart: unless-stopped
    ports:
    - 8232:8232 # RPC (security sensitive)
    - 8233:8233 # Zcash network (public)
    volumes:
    - zcash_data:/srv/zcashd/.zcash
    - params:/srv/zcashd/.zcash-params
  litewalletd:
    image: electriccoinco/lightwalletd
    depends_on:
      zcashd:
        condition: service_started
    command: >
      --grpc-bind-addr 0.0.0.0:9067
      --tls-cert /srv/lightwalletd/conf/fullchain1.pem
      --tls-key /srv/lightwalletd/conf/privkey1.pem
      --zcash-conf-path /srv/lightwalletd/conf/zcash.conf
      --data-dir /srv/lightwalletd/db_volume
      --log-file /dev/stdout
    restart: unless-stopped
    ports:
    - 9067:9067
    volumes:
    - litewalletd:/srv/lightwalletd/db_volume
    - zcash_litewalletd_conf:/srv/lightwalletd/conf
volumes:
  zcash_data:
    external: true
  params:
  litewalletd:
  zcash_litewalletd_conf:
    external: true

Per the docs on litewalletd, the mounted litewalletd volume will serve as a durable cache across recreations of the container. It will not store anything confidential or irreplaceable. It simply allows the litewalletd service to perform faster once the cache is populated. The litewalletd_conf volume on the other hand is for credentials, as I’ll elaborate on in the next section.

The --log-file switch makes docker logs -f zcash_litewalletd_1 work.

Setting up RPC credentials from litewalletd to zcashd

The litewalletd server permits credentials to be passed directly on the command line using --rpcuser and --rpcpassword, but this is a terrible idea. These credentials are extremely sensitive, as they unlock the ability to spend any ZEC you keep in that node’s wallet, among other things. By being present in the yml file, I would have to keep the yml file very secure. Keep in mind that on linux, the default permission even in your own home directory (?!) allows “others” read access to files.

Having the credentials in the yml file like would also mean that they appear in the command line of the container, which is accessible via docker inspect zcash_litewalletd_1, or whatever your container name is. It’s also accessible via ps a -e, even without running as root.

Given all these vulnerabilities, I took extra steps to avoid exposing credentials on the command line. Instead, I created a zcashd.conf file for my litewalletd server and passed the credentials through that. You can see in my .yml file how I mount a litewalletd_conf volume and then pass --zcash-conf-path to the server to point at the config file. Docker volumes are already protected so they require root access (or someone in the docker group at least).

Note that we have two zcashd.conf files:

1. zcashd itself requires one. This is stored in the root of the data volume.
2. litewalletd requires a different one.

Both files will contain the RPC credentials, but besides that, they are different. Here is the file I prepared for litewalletd:

rpcbind=zcashd
rpcuser=*redacted*
rpcpassword=*redacted*

This file must exist before the litewalletd container is created. That means I have to create the docker volume and populate it myself. In a shell with root access, I first created the volume. Note the volume name is as I prescribed it in the yaml file, with an added zcash_ prefix that came from the directory containing the yml file:

docker volume create zcash_litewalletd_conf

I now have to create or place the zcash.conf file inside that volume. You can find this volume in the host’s file system using this command:

docker volume inspect zcash_litewalletd_conf

For me, the path was:

/var/lib/docker/volumes/zcash_litewalletd_conf/_data

After I created the zcash.conf file under that directory, I set the ownership to match the UID used by the litewalletd container:

sudo chown 2002.2002 /var/lib/docker/volumes/zcash_litewalletd_conf/_data/zcash.conf

Exposed port security

The zcash.conf file used by the zcashd container must allow RPC access by the IP address used by the litewalletd container. Docker assigns IP addresses to containers with 172.*.*.* IP addresses, so I allowed all such addresses, as well as 127.0.0.1:

server=1
rpcuser=*redacted*
rpcpassword=*redacted*
rpcallowip=172.0.0.0/255.0.0.0
rpcallowip=127.0.0.1/255.255.255.255

If the zcashd container receives a request from an IP address outside the allowed ones, a 403 Forbidden response is returned.

All services in the docker-compose.yml file have access to all ports from all other services within that same file. That means you can avoid exposing your zcashd RPC port 8232 to the outside world if you have no intention of sending RPC requests directly to your zcashd node. The litewalletd service will still be able to reach it.

Setting up a service dependency

An optional step is to indicate in the docker-compose.yml file that the litewalletd container depends on the zcashd container. This gives docker-compose the insight to launch the zcashd container first, and tear it down last. It should also wait for zcashd to be ready to serve requests before starting the litewalletd container so that the litewalletd server doesn’t start by logging a bunch of failure messages and ultimately restart itself.

But the zcashd docker image does not come with a HEALTHCHECK in its Dockerfile, but fortunately docker gives us a way to elegantly enhance the zcashd docker image. To start, create a zcashd/Dockerfile file under the directory that contains the docker-compose.yml file with this content:

FROM electriccoinco/zcashd

HEALTHCHECK --interval=15s --start-period=3m CMD zcash-cli getblockchaininfo || exit 1

This leverages the zcash-cli tool that is included in the default image and runs the same RPC method that the litewalletd container starts with. By adding this healthcheck to the docker image, Docker itself can now discern between a ‘starting’ zcashd container and a healthy, running one.

We have a couple updates to make to the docker-compose.yml file now. First, update the zcashd service to build its own docker image instead of consuming the existing one:

-    image: electriccoinco/zcashd
+    build: ./zcashd

Then change the litewalletd‘s service dependency to reflect that we have a health check to honor:

 depends_on:
   zcashd:
-    condition: service_started
+    condition: service_healthy

Now take docker-compose down and back docker-compose up -d again. You’ll notice that zcashd is started first and sits there for a couple minutes before docker-compose then starts the litewalletd service. The first log entry from litewalletd is a success one instead of a string of several failures about zcashd loading the blockchain.

I did notice however that when rebooting the host, docker just restarts all services without any regard to the depends_on setting in docker-compose.yml. That’s unfortunate. I commented on a docker forum here in case we get any reply.

TLS security

I didn’t expect to have to set up TLS encryption for litewalletd because I have an nginx reverse-proxy on the network already and expected to just forward incoming HTTPS traffic from that to my HTTP litewalletd server. But here’s the secret I learned through network packet sniffing: litewalletd’s network protocol is not HTTP at all! All these URLs that are shown and accepted in these lite wallets that show litewalletd servers with the https: scheme are lying to you. The protocol is actually gRPC, and that means I can’t use nginx as a reverse proxy. Evidently the server can be launched with an HTTP server using the --http-bind-addr switch, but I haven’t tried it and the word is that many wallets may not support that.

And that means I have to expose the litewalletd port directly on the Internet (well, forwarded from my NAT router anyway).

Getting an HTTPS certificate from a trusted issuer is not hard. Since I already had one for nginx, I just reused the fullchain.pem and privkey.pem files obtained for that. I placed these files in the same litewalletd/conf directory as my zcash.conf file.

I actually did most of my litewalletd set up and testing without TLS. That is, instead of the two --tls switches, I passed in --no-tls-very-insecure and used http://localhost:9067 or http://hostname:9067 into the lite wallet software on my LAN to test it. Once that was proven to work, I changed my docker-compose.yml file to use the TLS secure settings, changed my DNS entries to point a new hostname at my public IP address, changed my router to forward the port to that particular machine, and I could then consume it with https://myhostname.mydomainname.com in my lite wallet client.

Note that once using https (which isn’t really HTTP at all, as I mentioned earlier) for the URL scheme, the lite wallet client authenticates the server’s SSL certificate, so using https://ip.add.res.s doesn’t work. It has to be a host name that matches the SSL cert.

As an aside, I really don’t see what the big deal is with encrypting litewalletd traffic. Lite wallets never transmit their private keys to the litewalletd server, which is really only trusted as far as validating the blockchain, and not with any secrets at all. So if the blockchain is public, and the server only serves as a middleman between full nodes and lite wallets, why is encryption relevant at all? My best guess is that because lite servers “know” which blocks a lite wallet is interested in, there is some minimal information disclosure that a lite wallet user may begrudgingly be willing to divulge to the lite server, but would rather not let the whole world in on.

But I am certainly not an expert on the threat analysis of lite servers. You may follow the discussion on this topic with the litewalletd owners.

Switching your lite wallet to your own server

At least some lite wallets allow you to switch from their own recommended lite server to a custom one. YWallet (Android) and ZecWallet Lite (Windows) are among these.

In my testing, ZecWallet Lite only lets you change the lite server after (ironically) fully syncing, which requires use of the default lite server. But you can get ahead of that and force it to use your local server by changing ~/.config/Zecwallet Lite/settings.json to point to your server:

{"all":{"lwd":{"serveruri":"http://localhost:9067"}}}

This is appropriate on the same machine as the litewalletd server. Obviously if it’s on another machine, you’ll need to adjust the URL.

On Windows, this file is found at %appdata%\Roaming\Zecwallet Lite\settings.json.

One satisfying way I tested this setting is that while ZecWallet Lite was doing the initial sync, between its CPU intensive ‘batches’, System Monitor would show huge spikes in network activity where the next batch was being downloaded. I used nmon to watch which network interfaces were being hit during those times, and found 2 were active. One was localhost, and the other related to docker. So indeed, ZecWallet Lite is now consuming my own litewalletd server. Yay.

I did notice however that ZecWallet Lite loses its ability to report USD value of Zcash when switching the Lightwallet Server to my custom one:

Troubleshooting

Diagnostic techniques

litewalletd logs can be reviewed with this command on the host OS:

docker logs -f zcash_litewalletd_1

zcashd IP filtering

One challenge I had was that although I could send RPC requests directly to the zcashd node, the litewalletd container evidently couldn’t, as it kept spewing errors in the log like this:

{"app":"lightwalletd","error":"status code: 403, response: \"\"","level":"warning","msg":"error with getblockchaininfo rpc, retrying…","retry":1,"time":"2022-12-13T04:14:25Z"}

It turns out that on the host machine, this succeeds when using the host machine’s IP address:

curl --user zecwallet -d '{"jsonrpc":"1.0","id":1,"method":"getblockchaininfo","params":[]}' -H 'Content-Type:text/plain' http://192.168.0.118:8232 -v

while using localhost fails:

curl --user zecwallet -d '{"jsonrpc":"1.0","id":1,"method":"getblockchaininfo","params":[]}' -H 'Content-Type:text/plain' http://localhost:8232 -v 

Evidently the use of the host’s DHCP-assigned IP address can be used to communicate with the zcashd server, but not localhost. And not the mycashd service name either, when tried from the litewalletd container.

This took a few hours to figure out. It ultimately was due to the allowed IP address filtering I had configured via the zcash.conf file. Evidently, the IP address used to access the port influences how the zcashd server sees what the source IP address is. My guess is the linux kernel is actually controlling this. I was able to resolve this by adding more allowed IP address ranges to the zcash.conf file, as I described earlier in this post.

Zecwallet Lite does not sync beyond block 1,732,500

Midway into syncing a new Zecwallet Lite installation (against the default Lite server), I terminated it and modified settings.json to force it to work against my lite server on localhost. This seemed to have brought sync to a halt. Although Zecwallet Lite didn’t complain, and on each launch it seemed to be trying to catch up to the tip of the blockchain, that only lasted a minute, and in the end it displayed 1732500 as the current block, no matter how many times I restarted the app.

When I tried switching Lite servers on an existing Zecwallet Lite installation on another machine that was already sync’d to tip, the app stopped launching, and I had to revert the settings.json change to get it to launch again.

Suspecting something wrong with my Litewalletd service, I tried using a REST client browser extension to send an HTTP request, similar to how I do for zcashd. But no matter what I send to localhost, I get a small response back full of spaces and one @ character. If I try it from the other machine, I get no response at all.

These symptoms continued even after the Litewalletd service itself had finished syncing to the current block, as evidenced by the docker container log.

Studying the lightwalletd README file revealed settings that were required in the zcash.conf that I provide to my zcashd node, that I was lacking. Specifically these:

txindex=1
insightexplorer=1
experimentalfeatures=1

I made those changes, ran docker-compose down to take down my existing containers, and manually reran the zcashd container with --reindex specified:

docker run --rm -v zcash_data:/srv/zcashd/.zcash -v zcash_params:/srv/zcashd/.zcash-params -p 8232:8232 -p 8233:8233 electriccoinco/zcashd --reindex 

This took a very long time. On my reasonably powerful machine, the first 90% of reindexing took 24 hours. The last 10% took 6 more days. I believe this is where the spammy transactions started on the block chain.

The lightwalletd docs mentioned that these settings would increase the size of my copy of the blockchain. They weren’t kidding. It shot up from around 176GB to 362GB.

When re-indexing completed, I stopped that container and used docker-compose up -d again to bring up both containers in their normal mode.