June 25, 2019 ademcan No comments

What happened to Saturn’s sixth moon last week?

TLDR; be sure that your node is using the additional storage to write the blockchain data there

As Enter Elastos we are running 3 Elastos supernodes (Titan, Ganymede and Callisto). To have a better overview of the ecosystem and be ready in case of issue we decided to use different approaches to run the different supernodes. There are 3 approaches available to run a supernode: docker, debian packages and binaries. Titan was running the node based on the binary approach. Yesterday, our server was complaining about not having enough space which resulted in the supernode going down and its status changing to inactive. Although an additional 100 GB of SSD was mounted, the system wasn’t using it. So, what happened?

If your run a supernode with the debian package approach, the blockchain data is written to the /data folder, and that’s where our additional SSD was mounted. However, if you use the binary approach, the data is saved to a relative path from where you run the node and not seeing the additional storage (if you have mounted it to /data).

What should you do in such situation?

1. If you are using the debian package approach

1.1. Format (before running the node) and mount the additional SSD to /data. Your SSD should be called /dev/xvdb, you can check with the following command:

lsblk

If you see /dev/xvdb in the list just continue the steps as follow, otherwise adapt the name of your additional storage in the following commands.

sudo mkdir /data
sudo mkfs -t xfs /dev/xvdb
sudo mount /dev/xvdb /data

1.2. Follow the automated script installation procedure

2. If you are using the binary approach

2.1. Format (before running the node) and mount the additional SSD to /home/ubuntu/supernode as an example. Your SSD should be called /dev/xvdb, you can check with the following command:

lsblk

If you see /dev/xvdb in the list just continue the steps as follow, otherwise adapt the name of your additional storage in the following commands. In the example below we will be running the node from the /home/ubuntu/supernode.

sudo mkdir /home/ubuntu/supernode
sudo mkfs -t xfs /dev/xvdb
sudo mount /dev/xvdb /home/ubuntu/supernode

2.2. Follow the binary supernode procedure from your /home/ubuntu/supernode folder

3. How to solve the issue

If your supernode was already running and you are facing the same issue, here is what to do:
1. Stop all the running service (ela, did, token and carrier)
2. Mount your additional storage and move all the data there
3. Restart all the services from the new location
4. Create a new activation transaction

./ela-cli wallet buildtx activate --nodepublickey $YOUR_NODE_PUBLIC_KEY

5. Send the activation transaction

 
./ela-cli wallet sendtx -f ready_to_send.txn --rpcuser $YOUR_RPC_USERNAME --rpcpassword $YOUR_RPC_PASSWORD

You should see your supernode coming back to life in a few minutes!

My main mistake here was not to check where the data was written to. As the REST API was working properly and the node was running fine I didn’t check the data/chain folder where the blockcahin is saved to. I just guessed it was all fine, it wasn’t, but now it is 🙂
I would like to thank Song (@songsjun) and his amazing team/colleagues that took their time to help me on a Sunday, as well as KP (@pachhaisaab) for putting all this information online and sharing his knowledge with the community. I am also somehow happy that this issue happened now (no penalty period) so that we are aware of it in case it happens to any other supernode.

Leave a Reply

Your email address will not be published. Required fields are marked *