运行“import-hbase.sh”文件以从 hbase 提取元数据时出现问题

Question

我正在使用以下容器运行 Docker 堆栈：

• 阿特拉斯（阿帕奇阿特拉斯）
• 风暴
• 卡桑德拉
• 雨云
• 卡夫卡
• hbase
• 动物园管理员

Apache Atlas 服务正在运行，我可以访问

localhost:21000

中的用户界面，就像 hbase (

localhost:16010

) 的情况一样。所有容器（Atlas、HBase、Zookeeper 等）都在同一个 Docker 网络中，并且可以相互通信。

但是当我在 atlas 中运行 import-hbase.sh 命令文件时，为了通过 atlas 摄取 hbase 元数据，我在将 atlas 连接到 hbase 时收到错误，这表明通过 Java 库连接 HBase 时出现问题：

2024-09-10 10:52:08,057 ERROR - [main:] ~ ImportHBaseEntities failed (HBaseBridge:198)
java.io.IOException: java.lang.reflect.InvocationTargetException
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:221)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114)
    at org.apache.hadoop.hbase.client.HBaseAdmin.available(HBaseAdmin.java:2353)
    at org.apache.atlas.hbase.bridge.HBaseBridge.<init>(HBaseBridge.java:216)
    at org.apache.atlas.hbase.bridge.HBaseBridge.main(HBaseBridge.java:158)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219)
    ... 4 more
Caused by: java.lang.NullPointerException
    at org.apache.hadoop.hbase.client.ConnectionImplementation.close(ConnectionImplementation.java:1939)
    at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:310)
    ... 9 more

我修改了 hbase-site.xml 文件，其中包含以下内容：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->

  <!--
    The following properties are set for running HBase as a single process on a
    developer workstation. With this configuration, HBase is running in
    "stand-alone" mode and without a distributed file system. In this mode, and
    without further configuration, HBase and ZooKeeper data are stored on the
    local filesystem, in a path under the value configured for `hbase.tmp.dir`.
    This value is overridden from its default value of `/tmp` because many
    systems clean `/tmp` on a regular basis. Instead, it points to a path within
    this HBase installation directory.

    Running against the `LocalFileSystem`, as opposed to a distributed
    filesystem, runs the risk of data integrity issues and data loss. Normally
    HBase will refuse to run in such an environment. Setting
    `hbase.unsafe.stream.capability.enforce` to `false` overrides this behavior,
    permitting operation. This configuration is for the developer workstation
    only and __should not be used in production!__

    See also https://hbase.apache.org/book.html#standalone_dist
  -->
<configuration>
  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>

  <property>
    <name>hbase.rootdir</name>
    <value>file:///tmp/hbase/data</value>
  </property>

  <property>
    <name>hbase.tmp.dir</name>
    <value>/tmp/hbase/data/tmp</value>
  </property>

  <property>
    <name>hbase.unsafe.stream.capability.enforce</name>
    <value>false</value>
  </property>

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>zookeeper</value>
  </property>

  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value> <!-- Puerto por defecto de Zookeeper -->
  </property>

  <!-- Configuración de puertos -->
  <property>
    <name>hbase.master.port</name>
    <value>16000</value>
  </property>
  <property>
    <name>hbase.master.info.port</name>
    <value>16010</value>
  </property>
  <property>
    <name>hbase.regionserver.port</name>
    <value>16020</value>
  </property>
  <property>
    <name>hbase.regionserver.info.port</name>
    <value>16030</value>
  </property> 

<!--  <property>
    <name>DIR</name>
    <value>/opt/hbase-2.5.6</value>
  </property>
  <property><name>DIR</name><value>/etc/hbase</value></property>
  <property><name>DIR</name><value>/etc/hbase</value></property>
  <property><name>DIR</name><value>/etc/hbase</value></property> 
<property><name>DIR</name><value>/etc/hbase</value></property>
<property><name>DIR</name><value>/etc/hbase</value></property> -->
</configuration>

有人可以帮我解决这个错误吗？

有人有从 hbase 或 kafka 等导入数据到 Apache Atlas 的成功经验吗？

Answer 1

hbase.rootdir

设置为

file:///tmp/hbase/data

。但由于您在 Docker 中运行分布式 HBase 集群，因此它可能需要指向分布式文件系统（HDFS 或 NFS）。

如果可能，将其更改为 HDFS 路径（例如，

hdfs://namenode:9000/hbase

）以与分布式集群设置保持一致。