ScPrime Documentation
Search…
'How-To' Linux Metadata Guide
This guide will help DIY Linux users to understand what ‘metadata’ is, why it’s so important to a provider, why you should be backing it up, common mistakes made etc.
When we refer to a provider’s metadata, we are referring to a folder with multiple components in it. Together they are your metadata and are probably the most vital part of a DIY provider.
By default on Linux, metadata is stored in the user’s home folder in a hidden folder called ‘.scprime’. If you’re running as root this would be /root/.scprime or if running as a normal user it would be /home/<USER>/.scprime. In this folder we can see above we have 6 folders plus a file. While you don’t really need to fully understand all the detail as to what all the files in the metadata do, it’s useful to have a high level understanding as that can help in troubleshooting issues relates to your metadata.
apipassword - this is a file with an API password which spc uses when using the spd API. Common errors we see related to this file are caused by performing some actions with a root account and some with a user account. Errors like:
[POST request error; API authentication failed.]
are caused by running spc commands with a different user than is running spd.
consensus - this folder contains the blockchain. When you first start up a provider node you either let spd download the blockchain automatically or you can download the bootstrap file from the software download page, unzip it into the consensus folder and then start up spd and it will sync the last few remaining blocks.
gateway - this folder contains a few files pertaining to other nodes in the network
host - this folder contains various files and folders specific to your provider like the provider settings which you would have setup like pricing, collateral, upload/download pricing etc., information about contracts
siamux - this is a legacy folder no longer used so we can safely ignore this one
transactionpool - this folder contains a database of all the transactions that have taken place for the provider
wallet - this folder contains a database of all your wallet transactions
Starting SPD and your Metadata
The most important concept to take away from this guide is that each time you startup spd, it needs to be told where the metadata folder is located that it’s going to use. If you use a default location then you don’t need to worry about specifying it (and this is the best approach as it has the least chance of going wrong). If you choose to use a non-default location then you must start spd every time with the -d flag and give it the location of your metadata. So for someone using the default location, the following command would work fine:
./spd -M gctwhHowever if you’ve chosen to store metadata in the /Metadata folder, then you must always start spd with the following command:
./spd -d /Metadata -M gctwhNow another important thing to note about Linux is paths are case sensitive so the following 3 paths are all different:
/Metadata
/MetaData
/metadata
So if you specify /MetaData when you were using /Metadata previously, you will create a new provider effectively. The best way to avoid this is to setup the autostart script as documented in the Linux Setup Guide here. Once you have this setup then the easiest way to start up spd correctly every time is to just reboot your Linux PC and let the script start spd in the same way every time.
Using an Incorrect or Non-existent Metadata Folder
If you do happen to start spd with either an incorrect path to your metadata or forget to specify it at all when using a non-default location, spd will startup but will start downloading a new consensus file and so you might see your Consensus not sync’d and a lower block height than expected as below:
You might also find when you try to unlock your wallet, you get an error like below:
Could not unlock wallet: [POST request error; error when calling /wallet/unlock: [wallet has not been encrypted yet; wallet has not been encrypted yet]]
This is another indication you have created a new provider as your correct wallet would be encrypted. Also looking at the output of
./spc host -v if you see your Provider ID doesn’t match what it was before. Or that you see no storage folders at the bottom of the output. These are all signs you have started spd with a new or different set of metadata. At this point you need to stop spc using ./spc stop and figure out where you have gone wrong as the situation is recoverable. If you for example add your storage folders back in, then you will lose any data in them and that won’t be recoverable.Once spd has been stopped and you’ve worked out where your real metadata is, startup spd again pointing at the correct location and you should see your original Provider ID, be able to unlock the wallet with your password or seed phrase and see your original storage folders. At this point it’s worth then deleting the incorrect set of metadata to avoid using it by mistake in the future.
If you are unsure of where your metadata might be or you’ve created a 2nd copy and don’t know where the right one is, use the following command to help find it:
sudo find / -name consensus.dbThis command will search your whole file system looking for consensus.db files, if you’ve got multiple sets of metadata then you will find multiple copies of the consensus.db file. For example:
[email protected]:~/ScPrime/ScPrime-v1.6.0-linux-amd64$ sudo find / -name consensus.db[sudo] password for scprime:find: ‘/run/user/1000/gvfs’: Permission denied/mnt/USB_Backup/metadata_backup/consensus/consensus.db/home/scprime/.scprime/consensus/consensus.dbHere we have one set of metadata in /mnt/USB_Backup/metadata_backup and another in /home/scprime/.scprime.
Where Should I Store My Metadata
As already mentioned above, the easiest thing to do is just leave it to be in its default location. This assumes that you’ve installed Linux onto an SSD and that the home folders are on that or another SSD. If you choose a non-default location, just ensure that it’s on an SDD drive. We do not recommend storing the metadata folder on a HDD, especially not the HDD used for your storage folder. The metadata is being written to a lot and needs fast read and write speeds for good performance overall of your provider.
Backing Up Your Metadata
It is highly recommended to backup your metadata folder. If anything goes wrong for example running out of disk space on the drive you are storing the metadata on then it can get corrupted and lead to spd not being able to load up and worse case you may have to start over with a new provider. A backup of your metadata can save this scenario from happening which in the future could result in a lot of lost data and therefore earnings.
It is recommended to backup at least once a day and setup an automated backup script to backup the metadata folder off the machine to either a server you have access to or at least a USB drive.
Assuming you have setup the auto startup script to start spd every time your PC reboots, it’s relatively easy to use that script as the basis for a backup script. If you’ve not setup the auto startup script it’s highly recommended to do that, documented in the Linux Setup Guide here.
For the backup script, we’ll create a file called metadata_backup.sh in the folder where spd and spc are located but it can be saved anywhere. Edit the file with Nano or Vi and paste in the following:
#!/bin/bashSPD_DATA=$HOME/.scprimeSCPRIME=$HOME/ScPrime/ScPrime-v1.6.0-linux-amd64# first stop spd$SCPRIME/spc stop# copy metadata folder to backup locationcp -r $SPD_DATA/* /mnt/USB_Backup/metadata_backup/# start spd in the same way as normalnohup $SCPRIME/spd --host-addr :44282 --host-api-addr :44283 -d $SPD_DATA -M gctwh &# unlock walletsleep 5sSCPRIME_WALLET_PASSWORD=`cat $HOME/ScPrime/.seed`export SCPRIME_WALLET_PASSWORD$SCPRIME/spc wallet unlockNote that your backup location, command to startup spd, metadata location and spd location will very likely be different, you will need to customize this script to your setup.
Save the file and then make it executable with the following command:
chmod 755 metadata_backup.shThe script basically stops spd from running, copies the metadata folder to our backup location, starts up spd and unlocks the wallet.
It’s important now to test the script so run it manually using the following:
./metadata_backup.shOnce it’s completed, check the backup folder and ensure you can see a copy of your metadata there and check to ensure spd is started up and your wallet is unlocked. If so, the final step is to create a cronjob to run the script on a regular basis. To do this, enter the following command:
crontab -eIf this brings a prompt to choose an editor, select your favorite one, that will open a file in your editor which is likely just some commented text with #’s at the start of each line. Go to the bottom of the file and enter the following:
@daily /home/scprime/ScPrime/ScPrime-v1.6.0-linux-amd64/metadata_backup.shSave the file and exit the editor. This sets the script to run daily. The path to where your script is located will likely differ so ensure you know the full path to your backup script. For the next few days, check your backups to ensure the backups are being taken and that spd is being restarted correctly.
Another option to backup your metadata is the following script written by discord community member Tacom Ogahiehe - https://github.com/Tacombel/scprime_metadata_backup
created 6/3/22 v1.0