如何在 Redshift 中创建日期表？

Question

tl;dr：我想在 Redshift 中生成一个日期表，以便更容易生成报告。最好不需要 Redshift 中已有的大型表，需要上传 csv 文件。

长版：我正在编写一份报告，其中我必须对一周中每天创建的新项目进行平均。日期范围可能跨越几个月或更长时间，因此可能有 5 个星期一，但只有 4 个星期日，这可能会使数学变得有点棘手。另外，我不能保证每天都有一个项目的实例，特别是当用户开始切片数据时。这会导致 BI 工具崩溃。

解决这个问题的最佳方法很可能是日期表。但是，大多数日期表教程都使用 Redshift 不可用或不完全支持的 SQL 命令（我正在看着你，generate_series）。

有没有一种简单的方法可以在 Redshift 中生成日期表？

我尝试使用的代码：（基于这个也不起作用的建议：http://elliot.land/post/building-a-date-dimension-table-in-redshift）

CREATE TABLE facts.dates (
  "date_id"              INTEGER                     NOT NULL PRIMARY KEY,

  -- DATE
  "full_date"            DATE                        NOT NULL,

  -- YEAR
  "year_number"          SMALLINT                    NOT NULL,
  "year_week_number"     SMALLINT                    NOT NULL,
  "year_day_number"      SMALLINT                    NOT NULL,

  -- QUARTER
  "qtr_number"           SMALLINT                    NOT NULL,

  -- MONTH
  "month_number"         SMALLINT                    NOT NULL,
  "month_name"           CHAR(9)                     NOT NULL,
  "month_day_number"     SMALLINT                    NOT NULL,

  -- WEEK
  "week_day_number"      SMALLINT                    NOT NULL,

  -- DAY
  "day_name"             CHAR(9)                     NOT NULL,
  "day_is_weekday"       SMALLINT                    NOT NULL,
  "day_is_last_of_month" SMALLINT                    NOT NULL
) DISTSTYLE ALL SORTKEY (date_id)
;


INSERT INTO facts.dates
(
   "date_id"
  ,"full_date"
  ,"year_number"
  ,"year_week_number"
  ,"year_day_number"

  -- QUARTER
  ,"qtr_number"

  -- MONTH
  ,"month_number"
  ,"month_name"
  ,"month_day_number"

  -- WEEK
  ,"week_day_number"

  -- DAY
  ,"day_name"
  ,"day_is_weekday"
  ,"day_is_last_of_month"
)
  SELECT
    cast(seq + 1 AS INTEGER)                                      AS date_id,

    -- DATE
    datum                                                         AS full_date,

    -- YEAR
    cast(extract(YEAR FROM datum) AS SMALLINT)                    AS year_number,
    cast(extract(WEEK FROM datum) AS SMALLINT)                    AS year_week_number,
    cast(extract(DOY FROM datum) AS SMALLINT)                     AS year_day_number,

    -- QUARTER
    cast(to_char(datum, 'Q') AS SMALLINT)                         AS qtr_number,

    -- MONTH
    cast(extract(MONTH FROM datum) AS SMALLINT)                   AS month_number,
    to_char(datum, 'Month')                                       AS month_name,
    cast(extract(DAY FROM datum) AS SMALLINT)                     AS month_day_number,

    -- WEEK
    cast(to_char(datum, 'D') AS SMALLINT)                         AS week_day_number,

    -- DAY
    to_char(datum, 'Day')                                         AS day_name,
    CASE WHEN to_char(datum, 'D') IN ('1', '7')
      THEN 0
    ELSE 1 END                                                    AS day_is_weekday,
    CASE WHEN
      extract(DAY FROM (datum + (1 - extract(DAY FROM datum)) :: INTEGER +
                        INTERVAL '1' MONTH) :: DATE -
                       INTERVAL '1' DAY) = extract(DAY FROM datum)
      THEN 1
    ELSE 0 END                                                    AS day_is_last_of_month
  FROM
    -- Generate days for 81 years starting from 2000.
    (
      SELECT
        '2000-01-01' :: DATE + generate_series AS datum,
        generate_series                        AS seq
      FROM generate_series(0,81 * 365 + 20,1)
    ) DQ
  ORDER BY 1;

这会引发此错误

[Amazon](500310) Invalid operation: Specified types or functions (one per INFO message) not supported on Redshift tables.;
1 statement failed.

...因为，我认为，在 Redshift 中的同一命令中不允许使用 INSERT 和 generate_series

Answer 1

问了这个问题，我就明白了。哎呀。

我从“事实”模式开始。

CREATE SCHEMA facts;

运行以下命令来启动数字表：

create table facts.numbers
(
  number int PRIMARY KEY
)
;

用它来生成您的号码列表。我花了一百万才开始

SELECT ',(' || generate_series(0,1000000,1) || ')'
;

然后将结果中的数字复制粘贴到下面的查询中的“值”之后：

INSERT INTO facts.numbers
VALUES
 (0)
,(1)
,(2)
,(3)
,(4)
,(5)
,(6)
,(7)
,(8)
,(9)
-- etc

^ 确保从复制粘贴的数字列表中删除前导逗号

一旦有了数字表，就可以生成日期表（再次从 elliot land 窃取代码

http://elliot.land/post/building-a-date-dimension-table-in-redshift）：

CREATE TABLE facts.dates (
  "date_id"              INTEGER                     NOT NULL PRIMARY KEY,

  -- DATE
  "full_date"            DATE                        NOT NULL,

  -- YEAR
  "year_number"          SMALLINT                    NOT NULL,
  "year_week_number"     SMALLINT                    NOT NULL,
  "year_day_number"      SMALLINT                    NOT NULL,

  -- QUARTER
  "qtr_number"           SMALLINT                    NOT NULL,

  -- MONTH
  "month_number"         SMALLINT                    NOT NULL,
  "month_name"           CHAR(9)                     NOT NULL,
  "month_day_number"     SMALLINT                    NOT NULL,

  -- WEEK
  "week_day_number"      SMALLINT                    NOT NULL,

  -- DAY
  "day_name"             CHAR(9)                     NOT NULL,
  "day_is_weekday"       SMALLINT                    NOT NULL,
  "day_is_last_of_month" SMALLINT                    NOT NULL
) DISTSTYLE ALL SORTKEY (date_id)
;


INSERT INTO facts.dates
(
   "date_id"
  ,"full_date"
  ,"year_number"
  ,"year_week_number"
  ,"year_day_number"

  -- QUARTER
  ,"qtr_number"

  -- MONTH
  ,"month_number"
  ,"month_name"
  ,"month_day_number"

  -- WEEK
  ,"week_day_number"

  -- DAY
  ,"day_name"
  ,"day_is_weekday"
  ,"day_is_last_of_month"
)
  SELECT
    cast(seq + 1 AS INTEGER)                                      AS date_id,

    -- DATE
    datum                                                         AS full_date,

    -- YEAR
    cast(extract(YEAR FROM datum) AS SMALLINT)                    AS year_number,
    cast(extract(WEEK FROM datum) AS SMALLINT)                    AS year_week_number,
    cast(extract(DOY FROM datum) AS SMALLINT)                     AS year_day_number,

    -- QUARTER
    cast(to_char(datum, 'Q') AS SMALLINT)                         AS qtr_number,

    -- MONTH
    cast(extract(MONTH FROM datum) AS SMALLINT)                   AS month_number,
    to_char(datum, 'Month')                                       AS month_name,
    cast(extract(DAY FROM datum) AS SMALLINT)                     AS month_day_number,

    -- WEEK
    cast(to_char(datum, 'D') AS SMALLINT)                         AS week_day_number,

    -- DAY
    to_char(datum, 'Day')                                         AS day_name,
    CASE WHEN to_char(datum, 'D') IN ('1', '7')
      THEN 0
    ELSE 1 END                                                    AS day_is_weekday,
    CASE WHEN
      extract(DAY FROM (datum + (1 - extract(DAY FROM datum)) :: INTEGER +
                        INTERVAL '1' MONTH) :: DATE -
                       INTERVAL '1' DAY) = extract(DAY FROM datum)
      THEN 1
    ELSE 0 END                                                    AS day_is_last_of_month
  FROM
    -- Generate days for 81 years starting from 2000.
    (
      SELECT
        '2000-01-01' :: DATE + number AS datum,
        number                        AS seq
      FROM facts.numbers
      WHERE number between 0 and 81 * 365 + 20
    ) DQ
  ORDER BY 1;

^ 请务必设置您需要的日期范围末尾的数字

Answer 2

作为解决方法，您可以在本地计算机上旋转 Postgres 实例，在那里运行代码，导出到 CSV，然后仅在 Redshift 中运行

CREATE TABLE

 部分并从 CSV 加载数据。由于这是一次性操作，所以可以这样做，这就是我实际上为新的 Redshift 部署所做的事情。

Answer 3

这里是构建不需要手动干预的

facts.numbers

 的不同建议：

Cross join

 该表本身足够多次以获得所需的行数

row_number() over (order by 1)

 将这些创建的记录转换为一组升序的数字

使用 Redshift 系统表的示例

pg_catalog.pg_operator

（截至 2020 年 10 月有 659 条记录）：

-- Prep, so that you can copy/paste the code sample
create schema if not exists facts;   -- Make sure the schema exists
drop table if exists facts.numbers;  -- Avoid an error if that table already exists;
create table facts.numbers           -- Create the table definition
(
  number int primary key
);

-- The bit you care about
insert into facts.numbers
    select     row_number() over (order by 1) -- return 1..n in place of the original record
    from       pg_catalog.pg_operator a       -- 659 records
    cross join pg_catalog.pg_operator b       -- to get 659^2=434k records 
    cross join pg_catalog.pg_operator c       -- to get 659^3=286M records
    limit      2000000                        -- to limit the result to a reasonable size
;

Answer 4

扩展上面的伟大想法 - 从一年中的第二天而不是第一天开始的小修复（BI 工具不应该对这种错过感到满意）+ 简化并修复标志 is_last_day_of_month：

CREATE SCHEMA IF NOT EXISTS dimensions;   -- Make sure the schema exists
DROP TABLE IF EXISTS dimensions.numbers;  -- Avoid an error if that table already exists;
CREATE TABLE dimensions.numbers           -- Create the table definition
(
  number INT PRIMARY KEY
);
-- Work around for Generate_series() and INSERT INTO by Sam Davey
INSERT INTO dimensions.numbers
    SELECT     row_number() over (order by 1) -- return 1..n in place of the original record
    FROM       pg_catalog.pg_operator a       -- 659 records
    CROSS JOIN pg_catalog.pg_operator b       -- to get 659^2=434k records
    CROSS JOIN pg_catalog.pg_operator c       -- to get 659^3=286M records
    LIMIT      1000000                        -- to limit the result to a reasonable size
; 
-- Elliot solution http://elliot.land/post/building-a-date-dimension-table-in-redshift
CREATE TABLE dimensions.dates (
  "date_id"              INTEGER                     NOT NULL PRIMARY KEY,

  -- DATE
  "full_date"            DATE                        NOT NULL,

  -- YEAR
  "year_number"          SMALLINT                    NOT NULL,
  "year_week_number"     SMALLINT                    NOT NULL,
  "year_day_number"      SMALLINT                    NOT NULL,

  -- QUARTER
  "qtr_number"           SMALLINT                    NOT NULL,

  -- MONTH
  "month_number"         SMALLINT                    NOT NULL,
  "month_name"           CHAR(9)                     NOT NULL,
  "month_day_number"     SMALLINT                    NOT NULL,

  -- WEEK
  "week_day_number"      SMALLINT                    NOT NULL,

  -- DAY
  "day_name"             CHAR(9)                     NOT NULL,
  "day_is_weekday"       SMALLINT                    NOT NULL,
  "day_is_last_of_month" SMALLINT                    NOT NULL
) DISTSTYLE ALL SORTKEY (date_id);

INSERT INTO dimensions.dates
(
   "date_id"
  ,"full_date"
  ,"year_number"
  ,"year_week_number"
  ,"year_day_number"

  -- QUARTER
  ,"qtr_number"

  -- MONTH
  ,"month_number"
  ,"month_name"
  ,"month_day_number"

  -- WEEK
  ,"week_day_number"

  -- DAY
  ,"day_name"
  ,"day_is_weekday"
  ,"day_is_last_of_month"
)
  SELECT
    CAST(seq + 0 AS INTEGER)                                      AS date_id,

    -- DATE
    datum                                                         AS full_date,

    -- YEAR
    CAST(EXTRACT(YEAR FROM datum) AS SMALLINT)                    AS year_number,
    CAST(EXTRACT(WEEK FROM datum) AS SMALLINT)                    AS year_week_number,
    CAST(EXTRACT(DOY FROM datum) AS SMALLINT)                     AS year_day_number,

    -- QUARTER
    CAST(TO_CHAR(datum, 'Q') AS SMALLINT)                         AS qtr_number,

    -- MONTH
    CAST(EXTRACT(MONTH FROM datum) AS SMALLINT)                   AS month_number,
    TO_CHAR(datum, 'Month')                                       AS month_name,
    CAST(EXTRACT(DAY FROM datum) AS SMALLINT)                     AS month_day_number,

    -- WEEK
    CAST(TO_CHAR(datum, 'D') AS SMALLINT)                         AS week_day_number,

    -- DAY
    TO_CHAR(datum, 'Day')                                         AS day_name,
    CASE WHEN TO_CHAR(datum, 'D') IN ('1', '7')
      THEN 0
    ELSE 1 END                                                    AS day_is_weekday,
    CASE WHEN LAST_DAY(datum) = datum THEN 1 ELSE 0 END           AS day_is_last_of_month
  FROM
    -- Generate days for 81 years starting from 2000.
    (
      SELECT
        ('2000-01-01' :: DATE - interval '1 day')::DATE + number AS datum,
        number                        AS seq
      FROM dimensions.numbers
      WHERE number between 0 and  81 * 365 + 20
    ) DQ
  ORDER BY 1;
DROP TABLE dimensions.numbers;

Answer 5

要解决创建事实.数字的问题，可以尝试这种更干净、更有效的方法。

create table facts.numbers
(number int PRIMARY KEY);

使用递归函数生成所需的数量，大约需要 60 秒生成 5 亿行。需要设置 max_recursion_rows 否则 redshift 会阻止你进行太深的递归。

SET SESSION max_recursion_rows = 500000000;
SHOW max_recursion_rows;

INSERT INTO facts.numbers
    WITH RECURSIVE number_series (number) AS (
      SELECT 1
      UNION ALL
      SELECT number + 1
      FROM number_series
      WHERE number < 500000000 -- Adjust the limit as per your desired number of rows
    )
    SELECT *
    FROM number_series;

如何在 Redshift 中创建日期表？

问题描述投票：0回答：5

5个回答

最新问题

如何在 Redshift 中创建日期表？

问题描述 投票：0回答：5

5个回答

最新问题

问题描述投票：0回答：5