最后是com.biao.flink.FlinkMain ,flink的入口类,也是Flink的流计算逻辑实现: publicclass FlinkMain {publicstaticvoidmain(String[] args)throwsException {//流式执行上下文,还有非流式的ExecutionEnvironment,缺省为Local模式,//远程的使用RemoteStreamEnvironment可在远程Flink集群上运行jar//RemoteStreamEnvironment remo...
开发Flink程序需要最低限度的API依赖,最低的依赖库包括flink-scala,flink-Streaming-scala。大多数应用需要依赖其他类库或连接器,例如kafka连接器,TableAPI,CEP库等,这些不是Flink核心依赖库的一部分,因此必须作为依赖项手动添加到应用程序中。 打开Flink官网中的,复制最低依赖项: https://ci.apache.org/projects/flin...
StreamSource表示的source端的operator,其既没有实现OneInputStreamOperator接口也没有实现TwoInputStreamOperator接口,由于其为流处理的源头,不需要接受输入 AbstractStreamOperator/AbstractUdfStreamOperator分析 AbstractStreamOperator是所有operator的基础抽象类,而AbstractUdfStreamOperator则是面向userFunction调用,接下来分析一下...
DataSource是flink数据的来源,用户可以通过env,addSource(SourceFunction)将SourceFunction添加到程序中,Flink内置许多已经实现的SourceFunction,但是用户也可以自定义实现SourceFunction(非并行化)接口,也可以实现ParallelSourceFunction(并行化)接口,如果有状态需要管理还可以实现RichParallelSourceFunction接口。 File Based readT...
StreamTask是流作业的任务基类,通常一个流作业的task启动由该方法的invoke函数为入口,本文基于Flink1.11.0该类生命流程进行分析。 StreamTask的构造 StreamTask的的初始化构造方法主要对一些参数进行设置,如configuration,stateBackend,timeService等 protectedStreamTask(Environmentenvironment,@NullableTimerServicetimerService,...
interface FlinkModel { String ENV_PARAM = "env"; String DEV_ENV = "dev"; String LOCAL_ENV_PARAM = "local"; /** * 获取resource下配置 * * @param envConf envConf * @return ParameterTool */ static ParameterTool getInitConfig(String envConf) { InputStream resourceAsStream = FlinkStream...
StreamTask.invoke public final void invoke() throws Exception { run(); } StreamTask是抽象基类,比如,OneInputStreamTask protectedvoidrun()throwsException {//cache processor reference on the stack, to make the code more JIT friendlyfinalStreamInputProcessor<IN> inputProcessor =this.inputProcessor;while...
2.2 StreamDataSource1 import org.apache.flink.api.java.tuple.Tuple3; import org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction; /** * Created by yidxue on 2018/9/12 */ public class StreamDataSource1 extends RichParallelSourceFunction<Tuple3<String, String, Long>> { ...
flink源码解析-flink获取streamGraph流图 2. 获取StreamGraph 3. 获取StreamGraph生成器 4. 生成StreamGraph 5. transform 5.1. StreamSource 5.1.1. streamGraph添加source 5.2. ContinuousFileReader 5.2.1. streamGraph添加Operator 5.2.2. streamGraph添加Edge 5.3. StreamMap 5.3.1. streamGraph添加Operator 5.3....
Flink Stream SQL Join Flink's Table API&SQL is an integrated query API for Scala and Java that allows the composition of queries from relational operators such as selection, filter, and join in an intuitive way. For details about Table API & SQL, visit the official website athttps://ci....