TLDR; be sure that your node is using the additional storage to write the blockchain data there
As Enter Elastos we are running 3 Elastos supernodes (Titan, Ganymede and Callisto). To have a better overview of the ecosystem and be ready in case of issue we decided to use different approaches to run the different supernodes. There are 3 approaches available to run a supernode: docker, debian packages and binaries. Titan was running the node based on the binary approach. Yesterday, our server was complaining about not having enough space which resulted in the supernode going down and its status changing to inactive. Although an additional 100 GB of SSD was mounted, the system wasn’t using it. So, what happened?
If your run a supernode with the debian package approach, the blockchain data is written to the /data folder, and that’s where our additional SSD was mounted. However, if you use the binary approach, the data is saved to a relative path from where you run the node and not seeing the additional storage (if you have mounted it to /data).
What should you do in such situation?
1. If you are using the debian package approach
1.1. Format (before running the node) and mount the additional SSD to /data. Your SSD should be called /dev/xvdb, you can check with the following command:
lsblk
If you see /dev/xvdb in the list just continue the steps as follow, otherwise adapt the name of your additional storage in the following commands.
sudo mkdir /data sudo mkfs -t xfs /dev/xvdb sudo mount /dev/xvdb /data
1.2. Follow the automated script installation procedure
2. If you are using the binary approach
2.1. Format (before running the node) and mount the additional SSD to /home/ubuntu/supernode as an example. Your SSD should be called /dev/xvdb, you can check with the following command:
lsblk
If you see /dev/xvdb in the list just continue the steps as follow, otherwise adapt the name of your additional storage in the following commands. In the example below we will be running the node from the /home/ubuntu/supernode.
sudo mkdir /home/ubuntu/supernode sudo mkfs -t xfs /dev/xvdb sudo mount /dev/xvdb /home/ubuntu/supernode
2.2. Follow the binary supernode procedure from your /home/ubuntu/supernode folder
3. How to solve the issue
If your supernode was already running and you are facing the same issue, here is what to do:
1. Stop all the running service (ela, did, token and carrier)
2. Mount your additional storage and move all the data there
3. Restart all the services from the new location
4. Create a new activation transaction
./ela-cli wallet buildtx activate --nodepublickey $YOUR_NODE_PUBLIC_KEY
5. Send the activation transaction
./ela-cli wallet sendtx -f ready_to_send.txn --rpcuser $YOUR_RPC_USERNAME --rpcpassword $YOUR_RPC_PASSWORD
You should see your supernode coming back to life in a few minutes!
My main mistake here was not to check where the data was written to. As the REST API was working properly and the node was running fine I didn’t check the data/chain folder where the blockcahin is saved to. I just guessed it was all fine, it wasn’t, but now it is 🙂
I would like to thank Song (@songsjun) and his amazing team/colleagues that took their time to help me on a Sunday, as well as KP (@pachhaisaab) for putting all this information online and sharing his knowledge with the community. I am also somehow happy that this issue happened now (no penalty period) so that we are aware of it in case it happens to any other supernode.