HBase Master always Stop...

Use Case

HBase Master Process always ERROR and stop the services after several minutes and need to start manually.

Solution

NTP Server not sync... So need to check NTP at where HBase Master and RegionalServer has been installed. So i found one of server that installed Regional Server has been stop.

[root@hdpworker1 ~]# ntpstat
synchronised to NTP server (103.16.182.23) at stratum 5
   time correct to within 262 ms
   polling server every 1024 s 

[root@hdpworker2 ~]# ntpstat
synchronised to NTP server (103.16.182.23) at stratum 5
   time correct to within 257 ms
   polling server every 1024 s 

[root@hdpworker3 ~]# ntpstat
timeout 

[root@hdpworker4 ~]# ntpstat
synchronised to NTP server (103.16.182.23) at stratum 5
   time correct to within 264 ms
   polling server every 1024 s

So need to start it again and restart all services for Hbase.

Refer :

• https://docs.couchbase.com/server/current/install/synchronize-clocks-using-ntp.html
• https://www.cyberciti.biz/faq/rhel-fedora-centos-configure-ntp-client-server/

But it still failed after 30 minutes HBase Master has been started. 

So i procced to 2nd step... What the 2nd Step? 

Solve issue with stale WAL splits, by removing WAL's from hdfs location /apps/hbase/data/WALs/ and restart the HBase master and region services.

1. Make a temporary file other than /apps/hbase/data/WALs/  by this commands

su -hdfs 

hdfs dfs -mkdir /apps/hbase/data/tempWALs

2. Why i create temporary folder? Because easy if anything happen..  then move all file under /apps/hbase/data/WALs to this directory.

hdfs dfs -mkdir /apps/hbase/data/WALs/* /apps/hbase/data/tempWALs/

3. Then restart all services. Hopefully this will your right solution.

For me, this has solve my problem.

Refer :

• https://community.cloudera.com/t5/Support-Questions/HBase-master-fails-to-start/td-p/146453
• https://techtalks.tech/knowledge-base/clear-dead-region-servers-in-hbase/