raw transaction Raw Transactions and Script

[buzzkillb]

I was looking for a way to clean up my inputs without going one by one in the QT and using coin control, when I stumbled upon an older post of someone asking how to do this on Bitcoin as this seems to happen frequently. The idea was take a bunch of small inputs (dust) and combine with a larger amount and send. Over time with a lot of transactions and staking the inputs get to become many and very small.

The basic way to create a raw transaction is by looking at your unspent list of transactions

listunspent

Which looks like this for my mining pool wallet which has a bunch of 0 fees after Denarius hit block 3mil 0 POW rewards and only mines tx fees.

Lets say I am going to create a new address [DRiQL7nc1zXmSaxVswutYu2CGZtsEBqfQQ] and send those 3 amounts to it.

We need to take note of the txid, vout, amount and tx fee for our coin. Denarius txfee is 0.00001 D

Line by line lets copy down what we need
f6d277fff890a86de7699e7ec8074e323556d89e895e652357da79c0d5fb77de 2 0.00000000
fa4c40deff3c9ac2eed8d20d21396465bcb19d4686583a1628b95e9a03f8ab4a 2 0.00000000
fb6e9ffdcb5b453e19fc7f46420faa329dd1ad1ec72699bdfa3f4856a24b362a 1 2.46936988
the total amount is 2.46936988 and then subtract 0.00001 & 0.00000001 for good measure, and send 2.46935987 D to our new address and combine these 3 inputs.

createrawtransaction "[{\"txid\":\"f6d277fff890a86de7699e7ec8074e323556d89e895e652357da79c0d5fb77de\",\"vout\":2},{\"txid\":\"fa4c40deff3c9ac2eed8d20d21396465bcb19d4686583a1628b95e9a03f8ab4a\",\"vout\":2},{\"txid\":\"fb6e9ffdcb5b453e19fc7f46420faa329dd1ad1ec72699bdfa3f4856a24b362a\",\"vout\":1}]" {\"DRiQL7nc1zXmSaxVswutYu2CGZtsEBqfQQ\":2.46935987}

Type this into the debug console or daemon line and adjust your txid's, vout's and at the end adjust your send to address and amount to send. Those \ seem to be required for this wallet to parse what you are sending in. This spits a transaction out.

01000000d3568d5e03de77fbd5c079da5723655e899ed85635324e07c87e9e69e76da890f8ff77d2f60200000000ffffffff4aabf8039a5eb928163a5886469db1bc656439210dd2d8eec29a3cffde404cfa0200000000ffffffff2a364ba256483ffabd9926c71eadd19d32aa0f42467ffc193e455bcbfd9f6efb0100000000ffffffff01b3f1b70e000000001976a914e1af17e42eba3528c9b9bd32e0df648e06eb0dcd88ac00000000

Now we want to sign the transaction from our wallet. signrawtransaction and the white output from above.

signrawtransaction 01000000d3568d5e03de77fbd5c079da5723655e899ed85635324e07c87e9e69e76da890f8ff77d2f60200000000ffffffff4aabf8039a5eb928163a5886469db1bc656439210dd2d8eec29a3cffde404cfa0200000000ffffffff2a364ba256483ffabd9926c71eadd19d32aa0f42467ffc193e455bcbfd9f6efb0100000000ffffffff01b3f1b70e000000001976a914e1af17e42eba3528c9b9bd32e0df648e06eb0dcd88ac00000000

This spits the signed info out.

{
"hex" : "01000000d3568d5e03de77fbd5c079da5723655e899ed85635324e07c87e9e69e76da890f8ff77d2f6020000006b483045022100f1ab0d5415c9f8f44edcdf493533f2ca538de43f18f994392fadb0f497d444af02200af35af9c9d01799a6fcab52fe8be8bb65c5fa16e1e17d6facdfecb78dc73ad00121020a6c9ce6f2a9ea59bb8f6fdd760c647c6cf86357e9e042645b9fc95c383fed54ffffffff4aabf8039a5eb928163a5886469db1bc656439210dd2d8eec29a3cffde404cfa020000006b483045022100cd09ee00e1965edbc8ffa084cf7b60b1ef0554426218c2ed5c864e4efdcffde102205f3a590631756d1fba2a597571fa805397741adae26fb708cb9138c1038332f50121020a6c9ce6f2a9ea59bb8f6fdd760c647c6cf86357e9e042645b9fc95c383fed54ffffffff2a364ba256483ffabd9926c71eadd19d32aa0f42467ffc193e455bcbfd9f6efb010000004847304402204c553ff24b9252986eacff676acbc43818e17570c06fd67ea707263d8cca037602205e032dc3a58ed9d4fb3f0fc9f20e2956f8459b68f712af69215d68c549b44f8301ffffffff01b3f1b70e000000001976a914e1af17e42eba3528c9b9bd32e0df648e06eb0dcd88ac00000000",
"complete" : true
}

Now for the send. sendrawtransaction and the long white out between the quotes.

sendrawtransaction 01000000d3568d5e03de77fbd5c079da5723655e899ed85635324e07c87e9e69e76da890f8ff77d2f6020000006b483045022100f1ab0d5415c9f8f44edcdf493533f2ca538de43f18f994392fadb0f497d444af02200af35af9c9d01799a6fcab52fe8be8bb65c5fa16e1e17d6facdfecb78dc73ad00121020a6c9ce6f2a9ea59bb8f6fdd760c647c6cf86357e9e042645b9fc95c383fed54ffffffff4aabf8039a5eb928163a5886469db1bc656439210dd2d8eec29a3cffde404cfa020000006b483045022100cd09ee00e1965edbc8ffa084cf7b60b1ef0554426218c2ed5c864e4efdcffde102205f3a590631756d1fba2a597571fa805397741adae26fb708cb9138c1038332f50121020a6c9ce6f2a9ea59bb8f6fdd760c647c6cf86357e9e042645b9fc95c383fed54ffffffff2a364ba256483ffabd9926c71eadd19d32aa0f42467ffc193e455bcbfd9f6efb010000004847304402204c553ff24b9252986eacff676acbc43818e17570c06fd67ea707263d8cca037602205e032dc3a58ed9d4fb3f0fc9f20e2956f8459b68f712af69215d68c549b44f8301ffffffff01b3f1b70e000000001976a914e1af17e42eba3528c9b9bd32e0df648e06eb0dcd88ac00000000

And then you get your new transaction id to check on the explorer.

1d9f74c8a791d004a83348b2350a9f25f67646f7baa632890ebe9e533275b359

Check on coinexplorer https://www.coinexplorer.net/D/transaction/1d9f74c8a791d004a83348b2350a9f25f67646f7baa632890ebe9e533275b359 and we can see the 3 inputs are now combined into a new address. Also those 0.00000000's disappear from our wallet.

[buzzkillb]

The above got me to thinking, how can I automate this? And that's where this script was born. https://gist.github.com/buzzkillb/878ff03068f141b01fa8afdcc92d9a25

What the script does is the user sets some parameters to look for in their wallet. Such as more than 1 input and a sum balance on an address of more than 0.001 D, and the script will search all of those addresses out, and combine every input in the address and send all of that back into the same address. Lets say you run a fortunastake and have 100's of 0.00000000 D inputs per address. This can help clear that up. Or you have been staking for a while and also mining, this can clear that up. If you run an exchange? Probably might work to clear the dust up. Basically does all of the steps above in a very crude bash script using jq, bc, awk, sed and some loops. All the user has to pay each time this is run is the tx fee per address.

[buzzkillb]

Using the script was okay. Which got me to reading about rust each night to figure out how to make a small utility. At the same time I wasn't quite sure how to even curl rpc the wallet from the very basic commands. Spent the day figuring out how to convert the dedust script into RPC calls. Which Windows also happens to have bash. Turn Windows Features on or off, and checkmark Windows Subsystem for Linux. Now you go to cmd.exe and then type bash. Pretty easy.

First curling RPC meant I could at least do these commands to a QT and not just the daemon. That's a plus. Then I saw bash works fine in Windows 10. That's a plus. Maybe works in MacOS too? Could be handy to try to create RPC commands for everything in the D wallet. What I wanted to do next was have a user config file (config.conf) so the user could change their rpcuser/rpcpass/port from 1 location and everything would run from that. Then I realized most likely you could just switch the port and work on any bitcoin fork. Pretty handy.

I got getbalance to work and was able to query my Windows QT and Ubuntu QT wallets. Off to figuring out the dedust script. I failed many many times trying to get the curl command to work in the bash script. But eventually I was able to get the command into a variable, then run the variable as a command. That seemed to get rid of all the weird escape codes, single quotes, double quotes, and every other oddity doing this in such a crude manner. And now someone has a way to dedust their many inputs back into the same address. Also since the code is so easy to read, anyone can figure out what the heck is going on. I was also trying to use less text files to read from, so I left just 1, raw.txt, which I might end up removing too. Hopefully someone finds this useful and will continue to try to integrate the different .sh files together to make an octopus of bash RPC commands all working together.

As of this writing I am calling the listunspent.sh from dedust.sh just to show what I am trying to do. Next would be how to call a createrawtransaction.sh and work my way down the whole process of creating, signing and sending a raw transaction.

https://github.com/buzzkillb/bash-denariusrpc

[buzzkillb]

Sample shown in Windows 10 powershell calling the Windows QT wallet.

[buzzkillb]

I got into thinking how could I make this more useful. Now playing around with a separate function script. Not sure if this useful yet, but it might be if I am trying to capture all the wallet commands through rpc. First I made a couple simple ones to see how they would work.

rpc.sh

#!/bin/bash
. config.conf

rpc_get_balance () {
curl --silent --data-binary '{"jsonrpc":"1.0","id":"curltext","method":"getbalance","params":[]}' -H 'content-type:text/plain;' http://"${rpcusername}":"${rpcpassword}"@"${rpchost}":"${rpcport}"/ | jq -r '.result'
}
rpc_get_blockcount () {
curl --silent --data-binary '{"jsonrpc":"1.0","id":"curltext","method":"getblockcount","params":[]}' -H 'content-type:text/plain;' http://"${rpcusername}":"${rpcpassword}"@"${rpchost}":"${rpcport}"/ | jq -r '.result'
}
rpc_get_info () {
curl --silent --data-binary '{"jsonrpc":"1.0","id":"curltext","method":"getinfo","params":[]}' -H 'content-type:text/plain;' http://"${rpcusername}":"${rpcpassword}"@"${rpchost}":"${rpcport}"/ | jq -r '.result'
}

getinfo.sh

#!/bin/bash
. config.conf
. rpc.sh
version=$(rpc_get_info | jq -r '.version')
protocolversion=$(rpc_get_info | jq -r '.protocolversion')
printf 'version: %s\n' "${version}"
printf 'protocolversion: %s\n' "${protocolversion}"

getinfo.sh looks for a config.conf and rpc.sh in the same directory. For now I have it parsing line by line the getinfo json from the curl rpc response. This example shows the version and protocolversion of the wallet.

Not sure the proper way to do this, but it works. Once I look at a few of these I want to look at how to pass variables into the function and bring them back out in their script. For instance how to createrawtransaction function using this idea for passing parameters into the params: [] box.

I am testing most of this on both Ubuntu 16.04, a little bit of 20.04, and Windows 10. The final goal would be interacting with influxdb or another database using just bash curl rpc to pass everything back and forth.

[buzzkillb]

Passing the functions through back into the script was confusing and it came down to 2 missing quotes on the signing of the rawtransaction that completely threw me off. Hint for me next time always use jq without the -r flag first and see what jq spits out. Signing requires quotes between the params []. Not 100% sure how to properly move back and forth between the dedust.sh and rpc.sh functions, but it appears to work so far. It didn't save me lines, but it did save me some confusion on how to get the curl command in. If anyone knows if I need to declare the variable with local in the function let me know, not sure.

rpc.sh

#!/bin/bash
. config.conf

rpc_get_balance () {
curl -s -d '{"jsonrpc":"1.0","id":"curltext","method":"getbalance","params":[]}' -H 'content-type:text/plain;' http://"${rpcusername}":"${rpcpassword}"@"${rpchost}":"${rpcport}"/ | jq -r '.result'
}
rpc_get_blockcount () {
curl -s -d '{"jsonrpc":"1.0","id":"curltext","method":"getblockcount","params":[]}' -H 'content-type:text/plain;' http://"${rpcusername}":"${rpcpassword}"@"${rpchost}":"${rpcport}"/ | jq -r '.result'
}
rpc_get_info () {
curl -s -d '{"jsonrpc":"1.0","id":"curltext","method":"getinfo","params":[]}' -H 'content-type:text/plain;' http://"${rpcusername}":"${rpcpassword}"@"${rpchost}":"${rpcport}"/ | jq -r '.result'
}
rpc_listunspent () {
curl -s -d '{"jsonrpc":"1.0","id":"curltext","method":"listunspent","params":[]}' -H 'content-type:text/plain;' http://"${rpcusername}":"${rpcpassword}"@"${rpchost}":"${rpcport}"/ | jq -r '.result'
}
rpc_createrawtransaction () {
createRawTransaction="curl -s -d '{\"jsonrpc\":\"1.0\",\"id\":\"curltext\",\"method\":\"createrawtransaction\",\"params\":["${1}"]}' -H 'content-type:text/plain;' http://"${rpcusername}":"${rpcpassword}"@"${rpchost}":"${rpcport}"/"
local createRawTransaction
}
rpc_signrawtransaction () {
signRawTransaction="curl -s -d '{\"jsonrpc\":\"1.0\",\"id\":\"curltext\",\"method\":\"signrawtransaction\",\"params\":["${1}"]}' -H 'content-type:text/plain;' http://"${rpcusername}":"${rpcpassword}"@"${rpchost}":"${rpcport}"/"
local signRawTransaction
}
rpc_sendrawtransaction () {
sendRawTransaction="curl -s -d '{\"jsonrpc\":\"1.0\",\"id\":\"curltext\",\"method\":\"sendrawtransaction\",\"params\":["${1}"]}' -H 'content-type:text/plain;' http://"${rpcusername}":"${rpcpassword}"@"${rpchost}":"${rpcport}"/"
local sendRawTransaction
}

dedust_fun.sh

#!/bin/bash
. config.conf
. rpc.sh

txfee=0.00010001
minimuminputs=1
minimumbalance=0.00100000

#functions used: rpc_listunspent rpc_createrawtransaction rpc_signrawtransaction rpc_sendrawtransaction

#deduplicate addresses
unspentaddresses=$(rpc_listunspent | jq -r '.[].address' | awk '!seen[$0]++')
while IFS= read -r
do
    printf '%s\n' $REPLY
    #count how many inputs per address
    txcount=$(rpc_listunspent | jq --arg ADDRESS "$REPLY" '[.[] | select(.address == $ADDRESS) | .txid ] | length')
    #echo "$txcount"
       #if address has minimuminputs then do this
       if [ "$txcount" -gt "$minimuminputs" ]; then
        echo "dedust addy: $REPLY"
        #get total balance per address
        sumdust=$(rpc_listunspent | jq --arg ADDRESS "$REPLY" '.[] | select(.address== $ADDRESS) | .amount | tonumber ' | jq -s add |  awk {' printf "%.8f",$1'})
        echo "denarii dust: $sumdust"
        subtractfee=$(echo "$sumdust - $txfee" | bc)
        #echo "$subtractfee"
           #check if minimumbalance is less than subtractfee
           if [ 1 -eq "$(echo "${minimumbalance} < ${subtractfee}" | bc)" ]
           then
     	      echo "Gonna send it!"
                rm raw.txt
		rpc_listunspent | jq --arg ADDRESS "$REPLY" '.[] | select(.address==$ADDRESS) | .txid,.vout ' |  (
		    while read txid; do
		        read vout
		        echo '{"txid":'$txid',"vout":'$vout'},' | tr -d ' \t\n\r\f' >> raw.txt
		    done
		)
                QUOTE="'"
                sed -i '1s/^/'$QUOTE''$QUOTE'[/' raw.txt
		sed -i '$ s/,$//g' raw.txt
		echo ']'$QUOTE''$QUOTE', '$QUOTE''$QUOTE'{"'$REPLY'":'$subtractfee'}'$QUOTE''$QUOTE'' | tr -d ' \t\n\r\f' >> raw.txt
		#store rawtransaction
    startrawtransaction=$(head -1 raw.txt)
		#put startrawtransaction json into curl function command
		rpc_createrawtransaction "${startrawtransaction}"
		#bring full curl rpc string into _createrawtransaction
		_createrawtransaction=$(eval "${createRawTransaction}")
		#RawTX builds the complete json from .result array
		RawTX=$(echo $_createrawtransaction | jq '.result')
		echo $RawTX

		rpc_signrawtransaction "$RawTX"
		_signrawtransaction=$(eval "${signRawTransaction}")
		RawSIGN=$(echo $_signrawtransaction | jq '.result.hex')
		echo $RawSIGN

		rpc_sendrawtransaction "$RawSIGN"
		_sendrawtransaction=$(eval "${sendRawTransaction}")
		echo $_sendrawtransaction
		RawSEND=$(echo $_sendrawtransaction | jq '.result')
		echo $RawSEND
		echo "combined dust minus fee: "$subtractfee
           fi
       fi
done <<< "$unspentaddresses"

For now I am just copy and pasting back and forth from gedit to github. I setup atom with shellcheck and a linter which will be my next step so I just connect to github and directly commit instead of this copy/pasta silliness. And probably should stop working on master branch, like how Denarius works on test branches before merging into Master and people think there is no dev work. I like the cleanliness of how Carsen does that large single commit for updates.

What I found was I needed to get my jibberish like the initial txid's and vouts and throw that into the curl function. So the raw.txt is created since I do want to see that text file for now, and puts that into a variable. Eventually I will convert this out and read a variable.

startrawtransaction=$(head -1 raw.txt)

From there I call the rpc_createrawtransaction with that startrawtransaction variable.

rpc_createrawtransaction "${startrawtransaction}"

I think this brings out the local variable I call out in the function itself createRawTransaction and now I evaluate this using eval. For some reason this string does not like to come in another way, something else I want to figure out what's going on. Now I have the _createrawtransaction variable to use which should be the full JSON the wallet sends us back which starts with a result array.

_createrawtransaction=$(eval "${createRawTransaction}")

Echoing the _createrawtransaction variable into JQ, I strip the result array and dig into what I want to use, the full code to throw into the signrawtransaction. Originally I was using jq -r .result.hex, and the -r was stripping the quotes I needed on the signing portion to get to sending. 

RawTX=$(echo $_createrawtransaction | jq '.result')

the last line in here echos RawTX variable since I want to see the feedback as I run this. Next is to convert all echos into printf and change variables into "${denariusrocks}" instead of all the variations thrown through here.

echo $RawTX

The rawtransaction functions themselves is just throwing properly formatted data into the "params":[ HERE ] area and running curl. Formatting is also important on the entire curl command, notice how this looks.

\"jsonrpc\":\"1.0\",\"id\":\"curltext\",\"method\":\"sendrawtransaction\",\"params\"

So now we have a rawtransaction rpc in bash, which I find very useful for automating sends. For now I am dedusting inputs. Who knows what other little things could be done all in little bash scripts.