Thursday, January 23, 2020

Should I merge or rebase in git?

A lot has been said about whether folks should rebase, squash or merge into their git repos. It has almost gotten to the level of religious arguments like tab vs. spaces.

On StackOverflow when the question came up, I provided this answer which included a link to Linus Torvalds' own take on the question. For those who are not aware, Linus invented git. I finished that StackOverflow answer with my own take on it that summarized Linus' take with my own flavor.

However since then, the forum where Linus stated his opinion has long since disappeared. I was able to recover it from the Wayback Machine, but as that site is slow and I want to be sure it's permanently archived, I copy the relevant forum post here for posterity:

From: Linus Torvalds linux-foundation.org>
Subject: Re: [git pull] drm-next
Newsgroups: gmane.comp.video.dri.devel
Date: 2009-03-29 21:45:45 GMT

On Sun, 29 Mar 2009, Dave Airlie wrote:
>
> My plans from now on are just to send you non-linear trees, whenever I 
> merge a patch into my next tree thats when it stays in there, I'll pull 
> Eric's tree directly into my tree and then I'll send the results, I 
> thought we cared about a clean merge history but as I said without some 
> document in the kernel tree I've up until now had no real idea what you 
> wanted.

I want clean history, but that really means (a) clean and (b) history.

People can (and probably should) rebase their _private_ trees (their own 
work). That's a _cleanup_. But never other peoples code. That's a "destroy 
history"

So the history part is fairly easy. There's only one major rule, and one 
minor clarification:

 - You must never EVER destroy other peoples history. You must not rebase 
   commits other people did. Basically, if it doesn't have your sign-off 
   on it, it's off limits: you can't rebase it, because it's not yours.

   Notice that this really is about other peoples _history_, not about 
   other peoples _code_. If they sent stuff to you as an emailed patch, 
   and you applied it with "git am -s", then it's their code, but it's 
   _your_ history.

   So you can go wild on the "git rebase" thing on it, even though you 
   didn't write the code, as long as the commit itself is your private 
   one.

 - Minor clarification to the rule: once you've published your history in 
   some public site, other people may be using it, and so now it's clearly 
   not your _private_ history any more.

   So the minor clarification really is that it's not just about "your 
   commit", it's also about it being private to your tree, and you haven't 
   pushed it out and announced it yet.

That's fairly straightforward, no?

Now the "clean" part is a bit more subtle, although the first rules are 
pretty obvious and easy:

 - Keep your own history readable

   Some people do this by just working things out in their head first, and 
   not making mistakes. but that's very rare, and for the rest of us, we 
   use "git rebase" etc while we work on our problems. 

   So "git rebase" is not wrong. But it's right only if it's YOUR VERY OWN 
   PRIVATE git tree.

 - Don't expose your crap.

   This means: if you're still in the "git rebase" phase, you don't push 
   it out. If it's not ready, you send patches around, or use private git 
   trees (just as a "patch series replacement") that you don't tell the 
   public at large about.

It may also be worth noting that excessive "git rebase" will not make 
things any cleaner: if you do too many rebases, it will just mean that all 
your old pre-rebase testing is now of dubious value. So by all means 
rebase your own work, but use _some_ judgement in it.

NOTE! The combination of the above rules ("clean your own stuff" vs "don't 
clean other peoples stuff") have a secondary indirect effect. And this is 
where it starts getting subtle: since you most not rebase other peoples 
work, that means that you must never pull into a branch that isn't already 
in good shape. Because after you've done a merge, you can no longer rebase 
you commits.

Notice? Doing a "git pull" ends up being a synchronization point. But it's 
all pretty easy, if you follow these two rules about pulling:

 - Don't merge upstream code at random points. 

   You should _never_ pull my tree at random points (this was my biggest 
   issue with early git users - many developers would just pull my current 
   random tree-of-the-day into their development trees). It makes your 
   tree just a random mess of random development. Don't do it!

   And, in fact, preferably you don't pull my tree at ALL, since nothing 
   in my tree should be relevant to the development work _you_ do. 
   Sometimes you have to (in order to solve some particularly nasty 
   dependency issue), but it should be a very rare and special thing, and 
   you should think very hard about it.

   But if you want to sync up with major releases, do a

 git pull linus-repo v2.6.29

   or similar to synchronize with that kind of _non_random_ point. That 
   all makes sense. A "Merge v2.6.29 into devel branch" makes complete 
   sense as a merge message, no? That's not a problem.

   But if I see a lot of "Merge branch 'linus'" in your logs, I'm not 
   going to pull from you, because your tree has obviously had random crap 
   in it that shouldn't be there. You also lose a lot of testability, 
   since now all your tests are going to be about all my random code.

 - Don't merge _downstream_ code at random points either.

   Here the "random points" comment is a dual thing. You should not mege 
   random points as far as downstream is concerned (they should tell you 
   what to merge, and why), but also not random points as far as your tree 
   is concerned.

   Simple version: "Don't merge unrelated downstream stuff into your own 
   topic branches."

   Slightly more complex version: "Always have a _reason_ for merging 
   downstream stuff". That reason might be: "This branch is the release 
   branch, and is _not_ the 'random development' branch, and I want to 
   merge that ready feature into my release branch because it's going to 
   be part of my next release".

See? All the rules really are pretty simple. There's that somewhat subtle 
interaction between "keep your own history clean" and "never try to clean 
up _other_ proples histories", but if you follow the rules for pulling, 
you'll never have that problem.

Of course, in order for all this to work, you also have to make sure that 
the people you pull _from_ also have clean histories.

And how do you make sure of that? Complain to them if they don't. Tell 
them what they should do, and what they do wrong. Push my complaints down 
to the people you pull from. You're very much allowed to quote me on this 
and use it as an explanation of "do this, because that is what Linus 
expects from the end result".

   Linus

Saturday, August 19, 2017

How to sell your Coinbase Multi-sig wallet coins on the Bitcoin Cash (BCH) network

The Bitcoin Cash (BCC/BCH) fork was controversial and many expected it to quickly die, which is why Coinbase announced ahead of the fork that customers would not have access to their Bitcoin Cash coins unless they withdrew their funds before the fork occurred. After the fork, Bitcoin Cash didn't die right away, and the price is sufficiently high that it lead customers to demand Coinbase change their plan, which they did by announcing customers will be able to sell their BCC by Jan 1, 2018.

But Jan 1 is a long way off, Bitcoin Cash might even be worthless by then and it's worth something now. Sadly if you had BTC in a standard vault or wallet with Coinbase, you must wait till Coinbase adds support to their site for you to access your BCH.

But for folks with multi-sig vaults set up by Coinbase for which they indicated they wanted to manage their own security, the good news is these folks can sell their BCH coins without delay. This is conditional on these users still possessing the key data they were given when they first set up the vaults, which is necessary to fill in their Multisig Vault recovery tool form. Since I had practiced extracting BTC myself using their tool, I knew I had the necessary key data, if I could only figure out how to apply it on the BCH network instead. This took a lot of work to figure out, and some help from others in the community, including some help that paid a small "bounty" for.

I share it with you since I am not alone in wanting to exchange my BCH. As it took quite a bit of work to derive these steps, and more to write it down in a consumable fashion for others, please consider donating some coin (BTC or BCH) to this address:

1Eft74zCo8ncqLWamgzpr418UtRTFPtaRL

Update 1/6/2018: Recent changes to network servers made these instructions difficult to follow in some circumstances. This blog post has been updated slightly so that these instructions work again.

Entry requirements

What you absolutely must have in order to complete these steps are control of 2 keys in your Coinbase multi-sig vault and knowledge of the 3rd public key. Coinbase would have shown you a page with this data:

  • User key
    • Seed
    • Public key
  • Shared key
    • Seed (encrypted)
    • Public key
  • Coinbase key
    • Public key

The shared key seed is encrypted with the password you gave Coinbase expressly for this vault. You must be in possession of that as well.

If you're missing any of the public keys, Coinbase will readily remind you of them if you visit your vault and click to view "Vault details". However if you lack the seeds, there does not appear to be a way to recover them, and you may have to just wait for Jan 1, 2018 with the rest of the folks at which point Coinbase should allow you to transfer them elsewhere with the 2 private keys it knows (so long as you remember your multisig vault password).

You do not have to understand the crypto behind bitcoin.

Steps to liberate your BCH

Important Disclaimer: Moving cryptocurrency around is an irreversible business, and this blog post takes you through manual steps using low-level tools. By a fault in following these steps, or a fault in the steps themselves as I have written them or the tools you'll use, your funds may be irreversibly lost. The steps I take you through and the tools I prescribe worked for me, and I expect they'll work for you to. But in sharing them in good faith I cannot take responsibility for loss of your coins.

One precaution I took the first time I did this, and I recommend you do as well, is to try it out using an address with only a small amount of coin that could possibly be lost. Confirm it is received on the other end, then go back and repeat the process for the addresses holding the larger amounts of coin. I'll point this out at an appropriate time down below as well.

Find the individual addresses, amounts, and their private keys

First some background on how these multi-sig vaults work is in order. The two keys you control are actually seeds rather than keys themselves. From these seeds are generated many other key pairs, and the coins in your vault are actually scattered across several addresses. We will need to identify which addresses actually contain your coin, then find the generated keys for each of those addresses based on the user and shared seeds you have.

Due to changes to network services you will need to download transactions from and broadcast to (that now require same-origin policy for AJAX calls), only FireFox with the CORS Everywhere plugin installed and active will allow you to complete the steps in this blog post.

To find the addresses with coin and derive the private keys for each, we will use my fork of Coinbase's own multisig-tool. My fork is only slightly different than coinbase's. I fixed a build break and then added an important line that will allow us to interactively script against the web page when it's done generating the data we need. (1/6/2018 Update: I also allowed the tool to work when your BTC balance is 0). This tool runs entirely on the client and you can download the pre-built web application. Unzip this file and open index.html with FireFox.

Now with your index.html page open, fill in the initial three public key fields and click "Continue". This will download information from the BTC network and populate your addresses and their individual balances. Many of the addresses may have a 0 balance and be gray, while a few will be black print with positive balances. These are the addresses we will need to collect individual private keys for.

But first, we need to provide the user and shared seed information. If you'd like you can first disconnect your computer from the Internet to assure yourself that this confidential information isn't being transmitted to a thief. Now go ahead and fill in the fields for the shared encrypted seed and the password to decrypt that shared seed. Also fill in the user seed. This tool also requires that you provide an address to send the coins to. We will not transmit coins with this tool, but go ahead and generate a valid BTC address and paste the public address into the tool. Or you could just go back to Coinbase and get a receiving address for one of your accounts and use that address here.

Now go ahead and click Continue. No need to worry about the transaction fee since again, you won't be broadcasting the created transaction. It will take a few seconds for the tool to decrypted the shared seed. When it is done, a very long hex code will be printed (which is the transaction we will not use) and below that is your (decrypted) shared seed. You won't need that either, but I added it to the print out of the tool for my own curiosity.

So if we won't use the transaction, nor the decrypted shared seed, what did we do all this for? The answer is for an object that the javascript in the web page generated that has a bunch of other information that you need. This 'object' is stored in a variable called 'document.vault', and is the real reason behind my own fork of this tool.

Hit F12 on your keyboard to bring up the Javascript debugger. Make sure you're looking at the debugger's "console" window. This is where you will type several javascript commands. It will help you understand what you're doing if you're a programmer, but I'll try to write these steps so that non-programmers can complete these instructions as well.

Now with your debugger console and the addresses with non-zero balances side-by-side, it should look something like this:


We're going to repeat several steps for each address non-zero balance. Note that the balances shown in this tool are your BTC funds -- not the balances on the BCH fork. If you haven't transferred any money out of your multi-sig vault since the fork, you can look at the balances reported by this recovery tool. Otherwise, you should copy and paste each address into https://blockdozer.com/ to see if it has a non-zero balance.

For each address with a non-zero balance we will look up several strings (i.e. sequences of characters) and you will need to record each of them. It is very important that you get each string recorded exactly, so you should copy and paste them into another file you create rather than typing them or writing them out by hand. Be careful how you store the data you collect and do not share it with anyone you don't trust, since it can be used to access your funds on the BTC and BCH networks.

  1. Determine the row # that the non-zero address appears on. In the example above, the first address with a non-zero balance is the third row. In Javascript counting starts at 0 instead of 1 so we say that the first non-zero address is at index=2 in the example above. In each command below, I will italicize the 2 that we obtained in this step. You should replace that italicized number with the actual number you come up with from your vault.
  2. Verify we counted right by printing the address in the javascript console using this command:
    document.vault.tx.addresses[2].address
    We should then see the same address printed in the javascript console that we found on the left. This is demonstrated in the above screenshot.
  3. Next we need to get the "redeem script" for this address using this same index. The redeem script is a long hex string. Print this redeem script with this command:
    document.vault.tx.addresses[2].redeemScript.toHex()Record this redeem script.
  4. Record the balance reported for this individual address. Note this is your BTC balance, which provided you haven't spent or received any funds to this vault since the fork should match your BCH balance at this address.
  5. This address has two private keys that you can derive for this address. Each are a 52 character string. Print them out with these commands and record their output:
    document.vault.tx.addresses[2].getPrvKey(document.vault.tx.hdWallets[0]).toWIF()
    document.vault.tx.addresses[2].getPrvKey(document.vault.tx.hdWallets[1]).toWIF()
After repeating these steps for each address with a non-zero balance, you now have all the inputs you need to create a transaction on the BCH network to spend your Bitcoin Cash!

Create and sign transactions to move your BCH to another address

Now that we have all the relevant redeem scripts and private keys to your BTC, we can use that same data to spend your coins on the Bitcoin Cash (BCC/BCH) network. To do this, we will use a fork of coinb.in, which an excellent open source project, again with all client-side javascript to keep your private keys safe. This tool lets you create and sign transactions on the BTC network. And dabura667's fork of it lets you do the same on the BCH network.

You will need to have a BCH address to send your Bitcoin Cash to. Presumably one that's easier to control via a BCH wallet application or a cryptocurrency exchange. Since I wanted to sell my BCH, I created an account on https://www.hitbtc.com/. Setting up this account was simple and immediate, and didn't even require an identity proof (IIRC). In addition, trades on this exchange have a low fee of 0.1%. I transferred my BCH there and sold them there for BTC. I then transferred my BTC to Coinbase where I could keep them, or sell them for USD. Wherever you get your BCH receiving address, keep it handy for later.

Download the BCH fork of coinb.in (diff) and open the index.html file in your favorite browser. We won't need to do any F12 debugging this time. We will use this local web application instead of the coinb.in web site on the Internet because BCH transactions are created and signed differently than BTC transactions.

Set the configuration to use Bitcoin Cash:

  1. Click New -> Transaction within the coinb.in app that's running on your local computer.
  2. Click to expand Advanced Options
  3. Under "Network", click the "settings" link.
  4. In the "Network" setting, select "Bitcoin Cash (mainnet)"
  5. Click "Submit Query"

You will also need to determine how much of a transaction fee to offer the miners for verifying your transaction. This is usually calculated for you automatically by most wallet apps including Coinbase, but since coinb.in is such a low-level application, we have to calculate it ourselves. The fee is based on the size of your transaction (not the amount transferred, but the physical size of the string that describes the transaction). Multi-sig transactions are larger than most other transactions and I found mine averaged around 350 bytes (i.e. characters). I used bitcoinfees.21.co to find the going rate for transactions at the time. If you use this site, be sure to switch it to show fees in BTC rather than "Satoshis" or any other unit. I couldn't find any site that showed fee averages for the BCH network, so I used this site to get an idea of what was normal, then I used Blockdozer Explorer to look at the latest block and picked a few transactions to see what the fee was there. Let's look at an example:


The above screenshot comes from bitcoinfees.21.co and tells me that I can expect to spend about 0.0000018 BTC per byte for each transaction I make. If I offer less than that to the miners, they may take significantly longer to verify the transaction, which means I wait longer to get my funds recognized as transferred at the receiving address.

Now let's take a look at a recent BCH block and see how that compares, over at Blockdozer:


I just picked a random transaction from the latest block. You would want to look at a transaction from a block that is recent by the time you're reading this. From the above screenshot, we see that a 192 byte transaction was accepted with a fee of 0.00092372 BCH. That works out to about 0.0000048 per byte. That's 3X more coin than the fee on the BTC network, but then BCH isn't worth as much as BTC so that seems about right. Let's go with this sample BCH transaction's fee rate. Whereas this sample transaction was 192 bytes long, we expect ours to be around 340 bytes, so we multiply to calculate our fee: 340 bytes * 0.0000048 = 0.001632 BCH, which today is worth about $1.61. That's the fee I would offer the miners if I were to create such a multi-sig transaction right now.

Now using the data you recorded in the previous section using the fork of the coinbase Multisig-tool, we will repeat several steps for each of the non-zero balance addresses you collected data for. A good precautionary technique would be to pick an address that has the lowest balance of all your addresses first, and wait for the whole transaction to be verified to confirm that everything works before you proceed with the addresses with the larger balances. So again, for each address you want to transfer funds from complete these steps:

  1. Connect to the Internet because once again, transactions history for your addresses will need to be downloaded.
  2. Click New -> Transaction again. If the "Advanced options" panel is expanded, collapse it now.
  3. Paste the redeem script that you recorded earlier into the first text box and click "Load".
  4. After the Inputs tab on the page updates to show a non-zero balance, you can disconnect your computer from the Internet if you wish in order to be comfortable that your private keys won't be transmitted to anyone else.
  5. Paste the BCH receiving address you obtained earlier into the "Address 1" field (I used one given by https://www.hitbtc.com/).
  6. Fill in the Amount to be transferred. Important: all funds at this address will be spent. The difference between the balance at that address and the amount you put in here will be your 'transaction fee' that you offer the miners for verifying your transaction. So you should calculate it like this: address balance - fee = Amount. So supposing you had 1.5 BCH at this address and the fee you were offering for this transaction was 0.001632 BCH, you would enter 1.5 - 0.001632 = 1.497984 into the Amount field here.
  7. This would be a great time to express your thanks for the research and time that went into this blog post so you could access your funds. If you feel so inclined, click the "+" symbol to the right of the "Amount" field to add another receiving address and paste in 1Eft74zCo8ncqLWamgzpr418UtRTFPtaRL and an amount you feel good about. Be sure to subtract that from the amount received by the first address so that the Transaction Fee remains unaffected.
  8. Double-check that the calculated Transaction Fee field just above the Submit button shows the transaction fee you intended (0.002016 in our example). If it does not, recalculate the Amount(s) for the receiving address so that their sum is equal to the balance at the sending address minus the intended transaction fee.
  9. Click Submit. This will not transmit the funds. But note the transaction string that is generated in the green panel. Note its size in bytes is displayed below it. This size is just the start. It will get larger when you sign it.
  10. Copy the hex string that is your Transaction to your clipboard.
  11. Click the Verify tab at the top of the web page. Paste your transaction into the box and click Submit. Review the receiving address and amount to double-check your work.
  12. Click the Sign tab at the top of the web page.
  13. Paste your (still unsigned) transaction into the big text box just above "Advanced options". 
  14. Now also from your notes, you should have two private keys associated with the sending address. copy and paste the first one into the "Private Key" text box above the transaction box.
  15. Click Submit. The transaction is signed with the first private key and reproduced in a green box.
  16. Verify that the signed transaction is longer than the original transaction. If there was a failure, the transaction will often simply be reprinted without a signature or any warning. You can detect this failure by seeing that the transaction length hasn't increased. If you find yourself in this case, double-check the steps you completed earlier and repeat till you get a longer transaction printed out.
  17. Now copy and paste this once-signed transaction from the green box into the transaction input box above it, replacing your original unsigned transaction.
  18. Replace the Private Key in the first text box with the second private key associated with the sending address that you recorded from the multisig-tool steps of the previous section.
  19. Click Submit again. This should sign the once-signed transaction with the second key, creating a twice-signed transaction that is longer in length than either of the prior two. Verify that it is longer. Mine is 339 bytes at this stage, which is very close to the 340 bytes I estimated my fee based on, so we can proceed. If your transaction size estimate is far off of this actual transaction length, you should go back to the step where you set the transaction fee, regenerate the transaction with a more appropriate fee and re-sign.
  20. Your transaction is now twice signed and ready for broadcast. Copy the signed transaction to your clipboard. Let's check your work one more time by click the Verify tab at the top of the web page and pasting your signed transaction into the text box and click Submit. Verify that the "Signed?" column shows it has been signed two times.
  21. Finally, to broadcast your transaction: with your twice signed transaction text on your clipboard, paste it into the text box of https://blockdozer.com/insight/tx/send and click "Send transaction". You may want to take note of the transaction ID so you can "follow" it until it receives a confirmation by the blockchain.
If you found this blog post helpful, please consider donating 1% of the BCH you were able to liberate to 1Eft74zCo8ncqLWamgzpr418UtRTFPtaRL, or any other amount of BCH or BTC to that same address you feel would appropriately express your thanks. Thank you, and enjoy reaping the benefits of the BCH fork!

Please also leave a comment indicating how this worked for you. If there were confusing/missing steps, etc. I will do my best to make corrections to this post based on the feedback I get. But if the steps work and you find yourself suddenly feeling richer, that would be great to hear as well.