Apache Airflow - A platform to programmatically author, schedule, and monitor workflows - airflow/Dockerfile.ci at master · brunobastosg/airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows - airflow/Dockerfile.ci at feature/aip-69-poc · jscheffl/airflow
ARG AIRFLOW_IMAGE_README_URL="https://raw.githubusercontent.com/apache/airflow/main/docs/docker-stack/README.md" # By default we install latest airflow from PyPI so we do not need to copy sources of Airflow # from the host - so we are using Dockerfile and copy it to /Dockerfi...
我用的是docker-airflow项目,项目地址:https://github.com/puckel/docker-airflow,该项目集成的最新airflow版本是1.10.1 airflow项目地址:https://github.com/apache/airflow,该项目最新版本是1.10.3 初步试水 首先我找了些中文的网站初步了解了下airflow,比如:https://segmentfault.com/a/1190000005835242 https:...
一、安装docker 下载docker安装包,下载地址:https://download.docker.com/linux/static/stable/x86_64/ 下载到本地后解压 tar -zxf docker-18.09.6.tgz 将解压出来的docker文件内容移动到 /usr/bin/ 目录下 cp docker/*/usr/bin/ 将docker注册为service ...
虽然是单点,但是配合mesos+hdfs nfs可以做成高可用用于生产环境; 参考: https://github.com/puckel/docker-airflow
(4)安装docker-compose curl -L "https://github.com/docker/compose/releases/download/1.25.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose 加上可执行权限: chmod +x /usr/local/bin/docker-compose (5)执行一下(在有docker-compose.yml的目录下面) ...
ARG AIRFLOW_IMAGE_README_URL="https://raw.githubusercontent.com/apache/airflow/main/docs/docker-stack/README.md" # By default we install latest airflow from PyPI so we do not need to copy sources of Airflow # from the host - so we are using Dockerfile and copy it to /Dockerfi...
s2i是source to image的缩写。简单地说,就是通过镜像生成镜像。github上的地址在这里。下面这个图,能很好地说明它的工作流程。 使用s2i后,就无须再为应用程序制作Dockerfile了,这些都交给s2i搞定即可。只需按s2i的要求,制作一个编译镜像Builder Image, 该镜像会从git仓库拉取应用代码,然后生成应用镜像App Image.应用...
除了这些基本的构建块之外,还有更多的特定处理器:DockerOperator,HiveOperator,S3FileTransferOperator,PrestoToMysqlOperator,SlackOperator 7、角色 webserver : 提供web端服务,以及会定时生成子进程去扫描对应的目录下的dags,并更新数据库 scheduler : 任务调度服务,根据dags生成任务,并提交到消息中间件队列中 (redis或ra...