Jump to content

joeuhren

Members
  • Content Count

    7
  • Joined

  • Last visited

  • Days Won

    2
  • Feedback

    N/A
  • BF$

    0 [ Donate ]

joeuhren last won the day on May 18

joeuhren had the most liked content!

Community Reputation

10 Good

About joeuhren

  • Rank
    Newbie
  1. Awesome guide! I'm looking forward to further simplifying and improving the seeder process in the future. Also, I figure I should move any further correspondence about the generic-seeder over to this thread since I sort of hijacked the Duct Tape DNS Seeder thread earlier. In response to your last question from the other thread: I haven't actually played much with tor and onion nodes before, so I can't give a definite YES, but the original sipa bitcoin-seeder has a single onion seed listed (kjy2eqzk4zwi5zd3.onion) and seems to have the necessary plumbing to handle onion nodes, I'm just not sure exactly how to use it. There is a cmd line argument that looks like it may help: -o <ip:port> Tor proxy IP/Port So I assume you need to setup and run a tor instance that is outside of the seeder - something like this https://www.linux.com/topic/networking/beginners-guide-tor-ubuntu/, and then you can run the seeder as per normal with the extra -o argument to pass your tor proxy ip and port so that the seeder can communicate with onion nodes. In theory I think that should be it. Someone please correct me if I am wrong because I'm assuming all of this just from a quick glance.
  2. There certainly may be an easier way to crawl a blockchain network, but I can't really think of any better way to do it since nodes on pretty much any blockchain are protected and will not communicate with you unless you send a properly formatted version msg to the node you want to communicate with. That being said, I've got some good and bad news in regards to adapting the dns seeder to the Denarius network: The Good News: I was successful in being able to crawl the Denarius network this weekend with the generic-seeder. It turns out that there are a few other important variables that need to be configured for proper communication with any altcoin network, and these values were never talked about or included in any of the earlier dns seeder setup guides. Many altcoins that I have looked at don't often change these variables and therefore the hardcoded values from the BTC network were working fine. Denarius is the first blockchain I have come across where some of these values were changed enough so that they broke communication with the default bitcoin-seeder values. The new config values that need to be changed are as follows: init_proto_version (Required): This is the protocol version that should be used as a starting value to communicate with nodes on the blockchain. Ex: 209. Typically you can find the init protocol version in your coins version.h file. min_peer_proto_version (Optional): This is the oldest/lowest protocol version that is allowed to communicate with nodes on the blockchain network. Ex: 70015. Typically you can find the minimum peer protocol version in your coins version.h file. Leave this value blank or set it to the same value as protocol_version if this setting does not exist in your blockchain. caddr_time_version (Required): This is the nTime value that is used to serialize CAddress data for the blockchain. Ex: 31402. Typically you can find the caddr_time_version in your coins version.h file. One thing to note is that Denarius was working with the bitcoin-seeder logic up until one of the last commits for v3.3.9.2 where the INIT_PROTO_VERSION and CADDR_TIME_VERSION values went above the hardcoded BTC values in the seeder which caused msgs sent from the seeder to be serialized slightly differently (using an older serialze method) which broke communication with the D wallets: https://github.com/carsenk/denarius/commit/e52995767bb526b5acc2c0172a586967f298c030#diff-c61070c281aed6ded69036c08bd08addR36 Here is the updated config that I am using in the generic-seeder to crawl the D network: protocol_version="33900" init_proto_version="33900" min_peer_proto_version="33900" caddr_time_version="33900" pchMessageStart_0 = "0xfa" pchMessageStart_1 = "0xf2" pchMessageStart_2 = "0xef" pchMessageStart_3 = "0xb4" wallet_port="33369" explorer_url="https://chainz.cryptoid.info/d/api.dws?q=getblockcount" second_explorer_url="" explorer_requery_seconds="60" block_count="3272984" seed_1="dnsseed.denarius.guide" seed_2="" seed_3="" seed_4="" seed_5="" seed_6="" seed_7="" seed_8="" seed_9="" seed_10="" cf_domain="" cf_domain_prefix="" cf_username="[email protected]" cf_api_key="" cf_seed_dump="dnsseed.dump" The Bad News: Even though I am able to crawl the Denarius network and it does find a handful of "good" nodes, it seems something may still be wrong with the communication between seeder and D nodes. I haven't had time to look into this properly yet as it took quite a while to get to this point. Basically what is happening is it seems to take a long time to get a response back from any and all D nodes and the vast majority of them are being discarded from the "good" list. I'll dig into this a bit more later as time permits, but I wanted to update you on the progress thus far.
  3. Ah ok, if you couldn't get the dnr-seeder working either then for sure there must be something specific to the way that denarius nodes talk to each other that isn't being handled in the seeder app. I'm quite curious as to what that could be, so I'll likely dig into it when I have the time and let you know what I find, but I can't promise when that will be exactly. Also, thanks for the note about the different package name in ubuntu 18.04. I didn't even realize because I'm behind the times and still doing everything in ubuntu 16.04, but it's on my list to start upgrading soon 😁
  4. I tried to crawl the Denarius network today, but I also couldn't get it to connect and didn't discover any new peers after a few minutes. However, I didn't do a deep dive to try to figure out what's wrong yet. For reference, here is the config I tried: protocol_version="33900" pchMessageStart_0 = "0xfa" pchMessageStart_1 = "0xf2" pchMessageStart_2 = "0xef" pchMessageStart_3 = "0xb4" wallet_port="33369" explorer_url="http://chainz.cryptoid.info/ric/api.dws?q=getblockcount" second_explorer_url="" explorer_requery_seconds="60" block_count="1280966" seed_1="dnsseed.denarius.guide" Do you know where I can find a copy of the existing seeder code that you might be using on the Denarius' network? I found and tried this one but I also couldn't get it to work properly: https://github.com/carsenk/dnr-seeder I even tried replacing some of the seed nodes in dnr-seeder but it didn't seem to make a difference. I imagine there must have been an update to the Denarius wallet/blockchain code that changes how nodes communicate with each other? I don't have a lot of time to dig into it right now, but let me know if you have any hints or ideas and I can try to make some time to look at it soon.
  5. Hey, that's an interesting problem that I hadn't thought of earlier. So interesting in fact that I posted a new commit today which should accomplish what you are asking. The logic was updated so that if the block height returned from the 1st explorer is the same as the last known block height, then it will try the 2nd explorer and choose the larger of the two values. I may consider adding a new option in the future that allows you to force checking the 2nd explorer every time regardless of the value from explorer 1, but I feel this change in behavior should cover most of the edge cases for now. Thanks for the suggested feature change! Let me know if you have any other suggestions or problems when you have a chance to take a proper look.
  6. I recently added a few new features to my generic-seeder project. New changes are as follows: Added a new config option for a 2nd explorer to be used as a fail-over in case the 1st explorer goes offline Added a new config option to specify how many seconds to wait between re-checking the explorer block height (previously it was hardcoded to 60 seconds) Added a new cmd line argument to force connections to IPv4 or IPv6 only if desired. Many blockchain networks are populated with nodes on either IPv4 or IPv6. There may be scenarios where one might want to find only the IPv4 nodes or only the IPv6 nodes on a blockchain network and this is this simplest way you could accomplish that. Added a step-by-step setup guide which details setting up the seeder using either cloudflare integration or the local DNS server source code and more info available here: https://github.com/team-exor/generic-seeder
  7. I'm sure its a long way from being "perfect", but take a look at the generic-seeder project I started earlier this year and see if it doesn't already do more-or-less what you are looking for: https://github.com/team-exor/generic-seeder I struggled way more than should have been necessary when I first started looking at how to adapt the bitcoin seeder to work with another coin. All the data that needs to change from coin to coin is buried in the source code in the most unobvious of places. So the first thing I did to remedy this was to add the ability to enter the kind of data that changes from coin to coin into a config file. That made it easy to quickly set up a new seeder for any coin in a matter of minutes (assuming you already know the current protocol version, port #, magic bytes, etc. for your coin). The next thing I struggled with was setting up the actual dns record to make proper use of the data that the crawler was collecting and dumping. I see you already have what looks like a great set of instructions to do the DNS setup here: https://denariustalk.org/index.php?/topic/314-dns-seeder-setup/. However, I did find that some savvy coder(s) figured out a much easier and fool-proof way to accomplish the same thing by utilizing a python script and retasking a cloudflare account into performing the DNS portion of the seeders work (as sort of mentioned above this post). In an ideal world, the code for this cloudflare setup should not be in a separate python script but should instead be part of the c++ seeder code to limit dependencies on other programming languages. I hope to either find the time to build this out more properly in the future or maybe some kind soul will do that for me given that it is open source after all. I did take the liberty of modifying the original cloudflare script(s) I found in the wild by fixing a few bugs and integrating the configuration for the python script into the same config file that is consumed by the generic-seeder app - so there's no need for separate config files all over the place. At this point, everything was working quite beautifully I thought, but still one thing bothered me about the original dns seeder code. I couldn't seem to understand why the block height that was being used to determine which nodes were considered "good" or "bad" was a static value. Now, I've been around a few blockchains over the years and one thing I've found that many chains suffer from are peers that are stuck on a block. To be honest, I never bothered to ask the original author of the bitcoin seeder why the block height check was based on a value that doesn't change. Perhaps there is a good reason for the way this was originally developed, and it could be that my next "fix" won't be appreciated or desired. But I decided that I wanted my seeder to effectively have the ability to weed out these pesky stuck peers and so the generic-seeder can also contact a trusted block explorer to achieve this goal. If a block explorer api url is specified in the config, the explorer api is contacted every 60 seconds (this should likely be a config variable for next update) and the updated block height value from the explorer is then used to validate against other nodes in the next series of tests. The block explorer integration is optional and the seeder still has the ability to hardcode the block height (using the config file) if you so choose. Feel free to clone or fork and build your own implementations on top of this. Again, it may certainly not be perfect, but I think it's a huge step in the right direction if I do say so myself. I hope you agree 😃
×
×
  • Create New...