Diary of ataicha: HBase Master

it all about me......

Recent

test
Showing posts with label HBase Master. Show all posts
Showing posts with label HBase Master. Show all posts

Monday, October 7, 2019

HBase Master always Stop...

Monday, October 07, 2019 0
HBase Master always Stop...
Use Case

HBase Master Process always ERROR and stop the services after several minutes and need to start manually.

Solution
NTP Server not sync... So need to check NTP at where HBase Master and RegionalServer has been installed. So i found one of server that installed Regional Server has been stop.
[root@hdpworker1 ~]# ntpstat
synchronised to NTP server (103.16.182.23) at stratum 5
   time correct to within 262 ms
   polling server every 1024 s 
[root@hdpworker2 ~]# ntpstat
synchronised to NTP server (103.16.182.23) at stratum 5
   time correct to within 257 ms
   polling server every 1024 s 
[root@hdpworker3 ~]# ntpstat
timeout 
[root@hdpworker4 ~]# ntpstat
synchronised to NTP server (103.16.182.23) at stratum 5
   time correct to within 264 ms
   polling server every 1024 s
So need to start it again and restart all services for Hbase.
But it still failed after 30 minutes HBase Master has been started. 
So i procced to 2nd step... What the 2nd Step? 
Solve issue with stale WAL splits, by removing WAL's from hdfs location /apps/hbase/data/WALs/ and restart the HBase master and region services.

1. Make a temporary file other than /apps/hbase/data/WALs/  by this commands
su -hdfs 
hdfs dfs -mkdir /apps/hbase/data/tempWALs
2. Why i create temporary folder? Because easy if anything happen..  then move all file under /apps/hbase/data/WALs to this directory.
hdfs dfs -mkdir /apps/hbase/data/WALs/* /apps/hbase/data/tempWALs/
3. Then restart all services. Hopefully this will your right solution.
For me, this has solve my problem.

Refer :