Flink 1.11 彻底的抛弃了这个机制，不再从 Query 来推断 PK 了，而是完全依赖 Create table 语法。比如 Create 一个 jdbc_table，需要在定义中显式地写好 Primary Key（后面 NOT ENFORCED 的意思是不强校验，因为 Connector 也许没有具备 PK 的强校验的能力）。当指定了 PK，就相当于就告诉框架这个Jdbc Sink ...
DROP TABLE IF EXISTS orders; CREATE TABLE IF NOT EXISTS orders ( order_id INT PRIMARY KEY NOT ENFORCED, order_date TIMESTAMP(0), customer_name STRING, product_id INT, price DECIMAL(10, 5), order_status BOOLEAN, region STRING ) WITH ( 'connector'='mysql-cdc', 'hostname'='xxx.xxx.xx...
CREATE TEMPORARY TABLE s1 ( a INT, b INT, ts as PROCTIME(), PRIMARY KEY (a) NOT ENFORCED ) WITH ( 'connector'='datagen', 'rows-per-second'='1', 'fields.b.kind'='random', 'fields.b.min'='0', 'fields.b.max'='10' ); CREATE TEMPORARY TABLE sink ( a BIGINT, b BIGINT )...
CREATETABLE`Order`(idINT,product_idINT,quantityINT,order_timeTIMESTAMP(3),PRIMARYKEY(id)NOTENFORCED
我们常见数据库的主键约束属于 PRIMARY KEY ENFORCED,会对数据进行校验。因为 Flink 并不持有数据,因此 Flink 支持的主键模式是 PRIMARY KEY NOT ENFORCED, 这意味着 Flink 不会校验数据,而是由用户确保主键的完整性。例如 HBase 里面对应的主键应该是 RowKey,在 MySQL 中对应的主键是在用户数据库的表中对应的主键...
create tableleft_upsert(id string,op_tstimestamp(3),primarykey(id)not enforced,watermarkforop_tsasop_ts-intervcal'0'second)with('connector'='upsert-kafka','properties.bootstrap.servers'='...','topic'='...''key.format'='json','value.format'='json','properties.group.id'='...') ...
(3),\n" +" SP_ID DECIMAL,\n" +" primary key (ACCT_ID) not enforced\n" +") WITH (\n" +" 'connector' = 'oracle-cdc',\n" +" 'hostname' = '***',\n" +" 'port' = '***',\n" +" 'username' = '***',\n" +" 'password' = '***',\n" +" 'database-name' ...
CREATE TABLE jdbc_table ( id BIGINT, ... PRIMARY KEY (id) NOT ENFORCED) 1. Hive 流批一体 首先看传统的 Hive 数仓。一个典型的 Hive 数仓如下图所示。一般来说,ETL 使用调度工具来调度作业,比如作业每天调度一次或者每小时调度一次。这里的调度,其实也是一个叠加的延迟。调度产生 table1,再产生 table...
='mysql-cdc','hostname'='197.XXX.XXX.XXX','port'='3306','username'='debezium','password'='PASSWORD','database-name'='cdcdb','table-name'='sbtest1','debezium.snapshot.mode'='initial');到DB：createtableprintSinkTable (idINT,kINT,cSTRING,padSTRING,primarykey (id) NOTENFORCED) with ...
s"""|CREATETABLEusers(|idint,|namestring,|PRIMARYKEY(id)NOTENFORCED|)|WITH(|'connector'='jdbc',|'url'='xxxx',|'driver'='$DRIVER_CLASS_NAME',|'table-name'='$tableName',|'lookup.cache.max-rows'='100',|'lookup.cache.ttl'='30s'...