{"id":1734,"date":"2020-06-24T17:51:35","date_gmt":"2020-06-24T09:51:35","guid":{"rendered":"http:\/\/39.97.231.10\/?p=1734"},"modified":"2020-06-24T17:51:35","modified_gmt":"2020-06-24T09:51:35","slug":"flink-on-zeppelin-batch","status":"publish","type":"post","link":"http:\/\/www.weijing.co\/?p=1734","title":{"rendered":"Flink on Zeppelin – Batch"},"content":{"rendered":"

\u672c\u6587\u8bb2\u8ff0\u5982\u4f55\u4f7f\u7528Flink SQL + UDF\u6765\u505aBatch ETL\u548cBI\u6570\u636e\u5206\u6790\u7684\u4efb\u52a1\u3002
\n<\/p>\n

Flink Interpreter\u7c7b\u578b<\/strong><\/section>\n

\u00a0 \u00a0 \u9996\u5148\u4ecb\u7ecd\u4e0bZeppelin\u4e2d\u7684Flink Interpreter\u7c7b\u578b\u3002Zeppelin\u7684Flink Interpreter\u652f\u6301Flink\u7684\u6240\u6709API (DataSet, DataStream, Table API \uff09\u3002\u8bed\u8a00\u65b9\u9762\u652f\u6301Scala\uff0cPython\uff0cSQL\u3002\u4e0b\u56fe\u662fZeppelin\u4e2d\u652f\u6301\u7684\u4e0d\u540c\u573a\u666f\u4e0b\u7684Flink Interpreter\u3002<\/p>\n\n\n\n\n\n\n\n\n
Name<\/strong><\/td>\nClass<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
%flink<\/td>\nFlinkInterpreter<\/td>\nCreates ExecutionEnvironment\/StreamExecutionEnvironment\/BatchTableEnvironment\/StreamTableEnvironment and provides a Scala environment<\/td>\n<\/tr>\n
%flink.pyflink<\/td>\nPyFlinkInterpreter<\/td>\nProvides a python environment<\/td>\n<\/tr>\n
%flink.ipyflink<\/td>\nIPyFlinkInterpreter<\/td>\nProvides an ipython environment<\/td>\n<\/tr>\n
%flink.ssql<\/td>\nFlinkStreamSqlInterpreter<\/td>\nProvides a stream sql environment<\/td>\n<\/tr>\n
%flink.bsql<\/td>\nFlinkBatchSqlInterpreter<\/td>\nProvides a batch sql environment<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\u914d\u7f6eFlink Interpreter<\/strong><\/section>\n

\u00a0 \u00a0 \u00a0 \u00a0\u4e0b\u56fe\u4f8b\u4e3e\u4e86\u6240\u6709\u91cd\u8981\u7684Flink\u914d\u7f6e\u4fe1\u606f\uff0c\u9664\u6b64\u4e4b\u5916\u4f60\u8fd8\u53ef\u4ee5\u914d\u7f6e\u4efb\u610fFlink\u7684Configuration\uff08https:\/\/ci.apache.org\/projects\/flink\/flink-docs-master\/ops\/config.html\uff09<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Property<\/strong><\/td>\nDefault<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
FLINK_HOME<\/td>\n<\/td>\nFlink\u7684\u5b89\u88c5\u76ee\u5f55<\/td>\n<\/tr>\n
HADOOP_CONF_DIR<\/td>\n<\/td>\nHadoop\u914d\u7f6e\u4fe1\u606f\u76ee\u5f55<\/td>\n<\/tr>\n
HIVE_CONF_DIR<\/td>\n<\/td>\nHive\u914d\u7f6e\u4fe1\u606f\u76ee\u5f55<\/td>\n<\/tr>\n
flink.execution.mode<\/td>\nlocal<\/td>\nExecution mode of flink, e.g. local | yarn | remote<\/td>\n<\/tr>\n
flink.execution.remote.host<\/td>\n<\/td>\njobmanager hostname if it is remote mode<\/td>\n<\/tr>\n
flink.execution.remote.port<\/td>\n<\/td>\njobmanager port if it is remote mode<\/td>\n<\/tr>\n
flink.jm.memory<\/td>\n1024<\/td>\nTotal number of memory(mb) of JobManager<\/td>\n<\/tr>\n
flink.tm.memory<\/td>\n1024<\/td>\nTotal number of memory(mb) of TaskManager<\/td>\n<\/tr>\n
flink.yarn.appName<\/td>\nZeppelin Flink Session<\/td>\nYarn app name<\/td>\n<\/tr>\n
flink.yarn.queue<\/td>\n<\/td>\nqueue name of yarn app<\/td>\n<\/tr>\n
flink.execution.jars<\/td>\n<\/td>\nadditional user jars (comma separated)<\/td>\n<\/tr>\n
flink.execution.packages<\/td>\n<\/td>\nadditional user packages (comma separated), e.g. org.apache.flink:flink-connector-kafka_2.11:1.10,org.apache.flink:flink-connector-kafka-base_2.11:1.10,org.apache.flink:flink-json:1.10<\/td>\n<\/tr>\n
zeppelin.pyflink.python<\/td>\npython<\/td>\npython binary executable for PyFlink<\/td>\n<\/tr>\n
table.exec.resource.default-parallelism<\/td>\n1<\/td>\nDefault parallelism for flink sql job<\/td>\n<\/tr>\n
zeppelin.flink.scala.color<\/td>\ntrue<\/td>\nwhether display scala shell output in colorful format<\/td>\n<\/tr>\n
zeppelin.flink.enableHive<\/td>\nfalse<\/td>\nwhether enable hive<\/td>\n<\/tr>\n
zeppelin.flink.printREPLOutput<\/td>\ntrue<\/td>\nPrint REPL output<\/td>\n<\/tr>\n
zeppelin.flink.maxResult<\/td>\n1<\/td>\nmax number of rows returned by sql interpreter<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

StreamExecutionEnvironment, ExecutionEnvironment, StreamTableEnvironment, BatchTableEnvironment<\/h2>\n

Flink Interpreter (%flink) \u4e3a\u7528\u6237\u81ea\u52a8\u521b\u5efa\u4e86\u4e0b\u97626\u4e2a\u53d8\u91cf\u4f5c\u4e3aFlink Scala\u7a0b\u5e8f\u7684\u5165\u53e3\u3002<\/p>\n