seeder Duct Tape DNS Seeder

[buzzkillb]

Spent some time with the DNS seeders and there is very little info so I was playing with cloudflare and wondered if I could automate the DNS seeds somehow. I assume this works on any coins that has a peer list with minor tweaks. The basic idea is getpeerinfo from the daemon into a json file and then send line by line of that into an A record on your seeder domain name. I am hopeful some others will see this and have a better idea how to automate this by making it easier to setup and run on generic coin.

https://github.com/buzzkillb/duct-tape-dns-seeder

Make a cloudflare account and point your domain denarius.pro at the cloudflare nameservers from your domain host control panel. Now we can edit records on cloudflare and the changes are almost immediate.

Cloudflare API Key is here, top right Icon -> My Profile -> View Global API Key

#Install Python Cloudflare

sudo apt install python-pip
git clone https://github.com/cloudflare/python-cloudflare
cd python-cloudflare
./setup.py build
sudo ./setup.py install

#Create a config file for your cloudflare API, change email and token (API KEY)

mkdir ~/.cloudflare
nano ~/.cloudflare/cloudflare.cfg

[CloudFlare]
email = <user@example.com>
token = <API KEY>
certtoken = v1.0-...
extras =

#test this works. change the ipv4 and denarius.pro to your stuff. dnsseed.denarius.pro is what my example will show.

cli4 --post name="dnsseed" type="A" content="73.218.220.108" /zones/:denarius.pro/dns_records

now we want to store a couple text files somewhere. you choose this for now I will use /root/

#create seed.sh and edit denarius.pro to your domain name. still using dnsseed.denarius.pro for this example.

#!/bin/sh
grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' "${1:?}" |
while read IP
do
    echo "$IP"
    cli4 --post name="dnsseed" type="A" content="$IP" /zones/:denarius.pro/dns_records
done

make this file executable

chmod +x seed.sh

now how to grab and put the ipv4's into the domain A records. denariusd daemon send peerinfo into a json file. then jq parses the json for the addr array and then we remove some junk and put that into peers.txt. From there the bash file uses regex to make the ipv4's pretty.

denariusd getpeerinfo > peer.json
jq '.[] | .addr' -r peer.json | sed 's/[][]//g' > peers.txt
./seed.sh peers.txt

This can be updated as much as the cloudflare API limits gives. But how to remove A records and try to keep this list fresh?

Lets make a delete file from python cloudflare examples. https://github.com/cloudflare/python-cloudflare/blob/master/examples/example_delete_zone_entry.py

#create delete.py and chmod+x this, and then put this inside.

#!/usr/bin/env python
"""Cloudflare API code - example"""

from __future__ import print_function

import os
import sys
import re
import json
import requests

sys.path.insert(0, os.path.abspath('..'))
import CloudFlare

def main():
    """Cloudflare API code - example"""

    try:
        zone_name = sys.argv[1]
        dns_name = sys.argv[2]
    except IndexError:
        exit('usage: example_delete_zone_entry.py zone dns_record')

    cf = CloudFlare.CloudFlare()

    # grab the zone identifier
    try:
        params = {'name':zone_name}
        zones = cf.zones.get(params=params)
    except CloudFlare.exceptions.CloudFlareAPIError as e:
        exit('/zones %d %s - api call failed' % (e, e))
    except Exception as e:
        exit('/zones.get - %s - api call failed' % (e))

    if len(zones) == 0:
        exit('/zones.get - %s - zone not found' % (zone_name))

    if len(zones) != 1:
        exit('/zones.get - %s - api call returned %d items' % (zone_name, len(zones)))

    zone = zones[0]

    zone_id = zone['id']
    zone_name = zone['name']

    print('ZONE:', zone_id, zone_name)

    try:
        params = {'name':dns_name + '.' + zone_name}
        dns_records = cf.zones.dns_records.get(zone_id, params=params)
    except CloudFlare.exceptions.CloudFlareAPIError as e:
        exit('/zones/dns_records %s - %d %s - api call failed' % (dns_name, e, e))

    found = False
    for dns_record in dns_records:
        dns_record_id = dns_record['id']
        dns_record_name = dns_record['name']
        dns_record_type = dns_record['type']
        dns_record_value = dns_record['content']
        print('DNS RECORD:', dns_record_id, dns_record_name, dns_record_type, dns_record_value)

        try:
            dns_record = cf.zones.dns_records.delete(zone_id, dns_record_id)
            print('DELETED')
        except CloudFlare.exceptions.CloudFlareAPIError as e:
            exit('/zones.dns_records.delete %s - %d %s - api call failed' % (dns_name, e, e))
        found = True

    if not found:
        print('RECORD NOT FOUND')

    exit(0)

if __name__ == '__main__':
    main()

to run the deleter, and it appears this only deletes 10-15 records at a time, so you might need to run this 5 times before sending a fresh list. This is only deleting records from dnsseed.denarius.pro. Nothing else on denarius.pro. Magical.

./delete.py denarius.pro dnsseed

Right now I am trying to think how frequent to send new ip's and delete the list and start over. Once I get that down I will post a sample cronjob to use. Otherwise this should work with basically any bitcoin fork daemon, maybe minor tweaks.

I also need a better regex to parse ipv6 so we can also make some on the fly AAAA records.

Use the github as that shows the crontab for adding and deleting the A records

[buzzkillb]

useful references

https://github.com/cloudflare/python-cloudflare

https://github.com/cloudflare/python-cloudflare/blob/master/cli4/cli4.man

https://github.com/cloudflare/python-cloudflare/blob/master/examples/example_delete_zone_entry.py

https://shapeshed.com/jq-json/

[buzzkillb]

https://github.com/buzzkillb/duct-tape-dns-seeder

[joeuhren]

I'm sure its a long way from being "perfect", but take a look at the generic-seeder project I started earlier this year and see if it doesn't already do more-or-less what you are looking for: https://github.com/team-exor/generic-seeder

I struggled way more than should have been necessary when I first started looking at how to adapt the bitcoin seeder to work with another coin. All the data that needs to change from coin to coin is buried in the source code in the most unobvious of places. So the first thing I did to remedy this was to add the ability to enter the kind of data that changes from coin to coin into a config file. That made it easy to quickly set up a new seeder for any coin in a matter of minutes (assuming you already know the current protocol version, port #, magic bytes, etc. for your coin).

The next thing I struggled with was setting up the actual dns record to make proper use of the data that the crawler was collecting and dumping. I see you already have what looks like a great set of instructions to do the DNS setup here: https://denariustalk.org/index.php?/topic/314-dns-seeder-setup/. However, I did find that some savvy coder(s) figured out a much easier and fool-proof way to accomplish the same thing by utilizing a python script and retasking a cloudflare account into performing the DNS portion of the seeders work (as sort of mentioned above this post). In an ideal world, the code for this cloudflare setup should not be in a separate python script but should instead be part of the c++ seeder code to limit dependencies on other programming languages. I hope to either find the time to build this out more properly in the future or maybe some kind soul will do that for me given that it is open source after all. I did take the liberty of modifying the original cloudflare script(s) I found in the wild by fixing a few bugs and integrating the configuration for the python script into the same config file that is consumed by the generic-seeder app - so there's no need for separate config files all over the place.

At this point, everything was working quite beautifully I thought, but still one thing bothered me about the original dns seeder code. I couldn't seem to understand why the block height that was being used to determine which nodes were considered "good" or "bad" was a static value. Now, I've been around a few blockchains over the years and one thing I've found that many chains suffer from are peers that are stuck on a block. To be honest, I never bothered to ask the original author of the bitcoin seeder why the block height check was based on a value that doesn't change. Perhaps there is a good reason for the way this was originally developed, and it could be that my next "fix" won't be appreciated or desired. But I decided that I wanted my seeder to effectively have the ability to weed out these pesky stuck peers and so the generic-seeder can also contact a trusted block explorer to achieve this goal. If a block explorer api url is specified in the config, the explorer api  is contacted every 60 seconds (this should likely be a config variable for next update) and the updated block height value from the explorer is then used to validate against other nodes in the next series of tests. The block explorer integration is optional and the seeder still has the ability to hardcode the block height (using the config file) if you so choose.

Feel free to clone or fork and build your own implementations on top of this. Again, it may certainly not be perfect, but I think it's a huge step in the right direction if I do say so myself. I hope you agree

[joeuhren]

I recently added a few new features to my generic-seeder project. New changes are as follows:

• Added a new config option for a 2nd explorer to be used as a fail-over in case the 1st explorer goes offline
• Added a new config option to specify how many seconds to wait between re-checking the explorer block height (previously it was hardcoded to 60 seconds)
• Added a new cmd line argument to force connections to IPv4 or IPv6 only if desired. Many blockchain networks are populated with nodes on either IPv4 or IPv6. There may be scenarios where one might want to find only the IPv4 nodes or only the IPv6 nodes on a blockchain network and this is this simplest way you could accomplish that.
• Added a step-by-step setup guide which details setting up the seeder using either cloudflare integration or the local DNS server

source code and more info available here: https://github.com/team-exor/generic-seeder

[buzzkillb]

Going to look at this, this week. Generic seeder sounds perfect. Denarius has 2 main block explorers and sometimes 1 or the other loses sync and gets stuck for a while. Can your failover allow it to check both and pick the higher height?

[joeuhren]

On 4/7/2020 at 8:24 PM, buzzkillb said:

Going to look at this, this week. Generic seeder sounds perfect. Denarius has 2 main block explorers and sometimes 1 or the other loses sync and gets stuck for a while. Can your failover allow it to check both and pick the higher height?

Hey, that's an interesting problem that I hadn't thought of earlier. So interesting in fact that I posted a new commit today which should accomplish what you are asking.

The logic was updated so that if the block height returned from the 1st explorer is the same as the last known block height, then it will try the 2nd explorer and choose the larger of the two values. I may consider adding a new option in the future that allows you to force checking the 2nd explorer every time regardless of the value from explorer 1, but I feel this change in behavior should cover most of the edge cases for now.

Thanks for the suggested feature change!

Let me know if you have any other suggestions or problems when you have a chance to take a proper look.

[buzzkillb]

On 4/9/2020 at 6:36 PM, joeuhren said:

Hey, that's an interesting problem that I hadn't thought of earlier. So interesting in fact that I posted a new commit today which should accomplish what you are asking.

The logic was updated so that if the block height returned from the 1st explorer is the same as the last known block height, then it will try the 2nd explorer and choose the larger of the two values. I may consider adding a new option in the future that allows you to force checking the 2nd explorer every time regardless of the value from explorer 1, but I feel this change in behavior should cover most of the edge cases for now.

Thanks for the suggested feature change!

Let me know if you have any other suggestions or problems when you have a chance to take a proper look.

Trying this out but not getting peers, any chance you can try with denarius? pchmessagestart here -> https://github.com/carsenk/denarius/blob/master/src/main.cpp#L4349

I gave it 5 peers to try and its stuck on 0/5 available.

[joeuhren]

15 hours ago, buzzkillb said:

Trying this out but not getting peers, any chance you can try with denarius? pchmessagestart here -> https://github.com/carsenk/denarius/blob/master/src/main.cpp#L4349

I gave it 5 peers to try and its stuck on 0/5 available.

I tried to crawl the Denarius network today, but I also couldn't get it to connect and didn't discover any new peers after a few minutes. However, I didn't do a deep dive to try to figure out what's wrong yet. For reference, here is the config I tried:

protocol_version="33900"
pchMessageStart_0 = "0xfa"
pchMessageStart_1 = "0xf2"
pchMessageStart_2 = "0xef"
pchMessageStart_3 = "0xb4"
wallet_port="33369"
explorer_url="http://chainz.cryptoid.info/ric/api.dws?q=getblockcount"
second_explorer_url=""
explorer_requery_seconds="60"
block_count="1280966"
seed_1="dnsseed.denarius.guide"

Do you know where I can find a copy of the existing seeder code that you might be using on the Denarius' network? I found and tried this one but I also couldn't get it to work properly: https://github.com/carsenk/dnr-seeder

I even tried replacing some of the seed nodes in dnr-seeder but it didn't seem to make a difference. I imagine there must have been an update to the Denarius wallet/blockchain code that changes how nodes communicate with each other? I don't have a lot of time to dig into it right now, but let me know if you have any hints or ideas and I can try to make some time to look at it soon.

[buzzkillb]

9 minutes ago, joeuhren said:

I tried to crawl the Denarius network today, but I also couldn't get it to connect and didn't discover any new peers after a few minutes. However, I didn't do a deep dive to try to figure out what's wrong yet. For reference, here is the config I tried:

protocol_version="33900"
pchMessageStart_0 = "0xfa"
pchMessageStart_1 = "0xf2"
pchMessageStart_2 = "0xef"
pchMessageStart_3 = "0xb4"
wallet_port="33369"
explorer_url="http://chainz.cryptoid.info/ric/api.dws?q=getblockcount"
second_explorer_url=""
explorer_requery_seconds="60"
block_count="1280966"
seed_1="dnsseed.denarius.guide"

Do you know where I can find a copy of the existing seeder code that you might be using on the Denarius' network? I found and tried this one but I also couldn't get it to work properly: https://github.com/carsenk/dnr-seeder

I even tried replacing some of the seed nodes in dnr-seeder but it didn't seem to make a difference. I imagine there must have been an update to the Denarius wallet/blockchain code that changes how nodes communicate with each other? I don't have a lot of time to dig into it right now, but let me know if you have any hints or ideas and I can try to make some time to look at it soon.

Also same issues. I wasn't able to get the dnr-seeder to find peers either. That's why I ended up making this thread with forcing peers into cloudflare.

Probably something specific to denarius then. From using your generic-seeder, you have made the best version of it for altcoins. Super easy to setup compared to all the other forks of sipa.

I forgot to write it down but libconfig++8-dev should be libconfig++-dev on ubuntu 18.04 to compile yours.

Since I like screenshots this is me trying to run it for about 12 hours on a test dns, I inputted 10 seeds initially.

[joeuhren]

3 hours ago, buzzkillb said:

Also same issues. I wasn't able to get the dnr-seeder to find peers either. That's why I ended up making this thread with forcing peers into cloudflare.

Probably something specific to denarius then. From using your generic-seeder, you have made the best version of it for altcoins. Super easy to setup compared to all the other forks of sipa.

I forgot to write it down but libconfig++8-dev should be libconfig++-dev on ubuntu 18.04 to compile yours.

Since I like screenshots this is me trying to run it for about 12 hours on a test dns, I inputted 10 seeds initially.

Ah ok, if you couldn't get the dnr-seeder working either then for sure there must be something specific to the way that denarius nodes talk to each other that isn't being handled in the seeder app. I'm quite curious as to what that could be, so I'll likely dig into it when I have the time and let you know what I find, but I can't promise when that will be exactly.

Also, thanks for the note about the different package name in ubuntu 18.04. I didn't even realize because I'm behind the times and still doing everything in ubuntu 16.04, but it's on my list to start upgrading soon

[buzzkillb]

16 hours ago, joeuhren said:

Ah ok, if you couldn't get the dnr-seeder working either then for sure there must be something specific to the way that denarius nodes talk to each other that isn't being handled in the seeder app. I'm quite curious as to what that could be, so I'll likely dig into it when I have the time and let you know what I find, but I can't promise when that will be exactly.

Also, thanks for the note about the different package name in ubuntu 18.04. I didn't even realize because I'm behind the times and still doing everything in ubuntu 16.04, but it's on my list to start upgrading soon

Yes please look into it whenever you have a chance. And realizing slowly its time to start testing everything on Ubuntu 18.04 and 20.04. Mainly 20.04 now since its so smooth.

Since your generic-seeder is the best implementation of this that I have seen, do you understand the basic way this crawls through found IP's? Maybe there is an even easier way to do this?

[joeuhren]

On 4/12/2020 at 2:24 PM, buzzkillb said:

Yes please look into it whenever you have a chance. And realizing slowly its time to start testing everything on Ubuntu 18.04 and 20.04. Mainly 20.04 now since its so smooth.

Since your generic-seeder is the best implementation of this that I have seen, do you understand the basic way this crawls through found IP's? Maybe there is an even easier way to do this?

There certainly may be an easier way to crawl a blockchain network, but I can't really think of any better way to do it since nodes on pretty much any blockchain are protected and will not communicate with you unless you send a properly formatted version msg to the node you want to communicate with.

That being said, I've got some good and bad news in regards to adapting the dns seeder to the Denarius network:

The Good News:

I was successful in being able to crawl the Denarius network this weekend with the generic-seeder. It turns out that there are a few other important variables that need to be configured for proper communication with any altcoin network, and these values were never talked about or included in any of the earlier dns seeder setup guides. Many altcoins that I have looked at don't often change these variables and therefore the hardcoded values from the BTC network were working fine. Denarius is the first blockchain I have come across where some of these values were changed enough so that they broke communication with the default bitcoin-seeder values.

The new config values that need to be changed are as follows:

• init_proto_version (Required): This is the protocol version that should be used as a starting value to communicate with nodes on the blockchain. Ex: 209. Typically you can find the init protocol version in your coins version.h file.
• min_peer_proto_version (Optional): This is the oldest/lowest protocol version that is allowed to communicate with nodes on the blockchain network. Ex: 70015. Typically you can find the minimum peer protocol version in your coins version.h file. Leave this value blank or set it to the same value as protocol_version if this setting does not exist in your blockchain.
• caddr_time_version (Required): This is the nTime value that is used to serialize CAddress data for the blockchain. Ex: 31402. Typically you can find the caddr_time_version in your coins version.h file.

One thing to note is that Denarius was working with the bitcoin-seeder logic up until one of the last commits for v3.3.9.2 where the INIT_PROTO_VERSION and CADDR_TIME_VERSION values went above the hardcoded BTC values in the seeder which caused msgs sent from the seeder to be serialized slightly differently (using an older serialze method) which broke communication with the D wallets: https://github.com/carsenk/denarius/commit/e52995767bb526b5acc2c0172a586967f298c030#diff-c61070c281aed6ded69036c08bd08addR36

Here is the updated config that I am using in the generic-seeder to crawl the D network:

protocol_version="33900"
init_proto_version="33900"
min_peer_proto_version="33900"
caddr_time_version="33900"
pchMessageStart_0 = "0xfa"
pchMessageStart_1 = "0xf2"
pchMessageStart_2 = "0xef"
pchMessageStart_3 = "0xb4"
wallet_port="33369"
explorer_url="https://chainz.cryptoid.info/d/api.dws?q=getblockcount"
second_explorer_url=""
explorer_requery_seconds="60"
block_count="3272984"
seed_1="dnsseed.denarius.guide"
seed_2=""
seed_3=""
seed_4=""
seed_5=""
seed_6=""
seed_7=""
seed_8=""
seed_9=""
seed_10=""
cf_domain=""
cf_domain_prefix=""
cf_username="[email protected]"
cf_api_key=""
cf_seed_dump="dnsseed.dump"

The Bad News:

Even though I am able to crawl the Denarius network and it does find a handful of "good" nodes, it seems something may still be wrong with the communication between seeder and D nodes. I haven't had time to look into this properly yet as it took quite a while to get to this point. Basically what is happening is it seems to take a long time to get a response back from any and all D nodes and the vast majority of them are being discarded from the "good" list. I'll dig into this a bit more later as time permits, but I wanted to update you on the progress thus far.

[buzzkillb]

52 minutes ago, joeuhren said:

The Bad News:

Even though I am able to crawl the Denarius network and it does find a handful of "good" nodes, it seems something may still be wrong with the communication between seeder and D nodes. I haven't had time to look into this properly yet as it took quite a while to get to this point. Basically what is happening is it seems to take a long time to get a response back from any and all D nodes and the vast majority of them are being discarded from the "good" list. I'll dig into this a bit more later as time permits, but I wanted to update you on the progress thus far.

Thank you for this, you are reading my mind. As I started to look at constructing raw transactions to broadcast onto electrumx, I started to look into just broadcasting those out to peers instead and found the most simple version someone made to do this using python. I figured if that was having issues pinging the nodes, there must be another issue. Carsen is currently trying to find it, but I also remember the person running the seeder worked up until 3.3.9.2 version, so this is good news, as probably Carsen can pin this down what changed. 

Funny enough looking for that python thing I saw your original post on bitcointalk for the seeder. I don't think people understand how easy you made the original sipa one to setup. Extremely useful DNS seeder.

[buzzkillb]

Testing out your latest build and many thanks for modifying this to work. Seeing the dnsseed.dump have IP's made my day. Now I can start building some stuff from that list. Does this handle onion nodes?