如何获取数据块工作区中所有表和数据库的名称

Question

我想在单个数据框或表或视图中查看数据库名称、表名称、配置单元的路径

table_info = []
for table in tables:
    table_info.append({"Database": table.database, "Table": table.name, "Location": table.location})

df = spark.createDataFrame(table_info)

Answer 1

我已经详细解释了这里的步骤：

https://medium.com/@debayankar/getting-information-schema-details-of-databricks-6c7651e6184b

这是使用的代码：

注意：我想查看具有特定前缀的数据库

from pyspark.sql import SparkSession, Row

# create a SparkSession
spark = SparkSession.builder.appName("ShowTablesInfo").getOrCreate()

# create an empty list to hold the DataFrames
df_list = []

# get all databases in the workspace that start with "edap"
databases = [database.name for database in spark.catalog.listDatabases() if database.name.startswith("edap")]

# loop through each database and retrieve the table information
for database in databases:
    print(f"Tables in database {database}:")

    # set the current database
    spark.catalog.setCurrentDatabase(database)

    # check if there are tables in the database
    if len(spark.catalog.listTables()) == 0:
        print("No tables found in the database.")
    else:
        # get all tables
        tables = spark.catalog.listTables()

        # create a list of dictionaries containing the table information
        table_info = []
        for table in tables:
            if table.tableType == 'MANAGED' or table.tableType == 'EXTERNAL':
                name = table.name
                location = spark.sql(f"DESCRIBE EXTENDED {name}").filter("Location").select("data_type").collect()[0][0]
                table_info.append({"Database": database, "Table": name, "Location": location})

        # create a DataFrame from the list of dictionaries
        df = spark.createDataFrame([Row(**x) for x in table_info])

        # add the DataFrame to the list
        df_list.append(df)

# concatenate the DataFrames in the list
if len(df_list) > 0:
    df_combined = df_list[0]
    for i in range(1, len(df_list)):
        df_combined = df_combined.union(df_list[i])

    # show the combined DataFrame
    df_combined.show()
else:
    print("No tables found in any database.")

# stop the SparkSession
spark.stop()

输出

+-----------+-----------------+----------------------------------+
| Database  | Table           | Location                         |
+-----------+-----------------+----------------------------------+
| edap_demo | sales           | /mnt/sales_data                  |
| edap_demo | customers       | /mnt/customer_data               |
| edap_demo | products        | /mnt/product_data                |
| edap_logs | server_logs     | /mnt/log_data/server_logs        |
| edap_logs | application_logs| /mnt/log_data/application_logs   |
+-----------+-----------------+----------------------------------+