一、环境准备 hadoop 2.10.1 hive 2.3.6 hudi 0.9 flink 1.12.2 各组件的安装配置就不介绍了，可以google搜索下就能找到 二、flink+hudi实战步骤 flink 单机或者集群配置好，下载hudi依赖包 wget https://repo.maven.apache.org/maven2/org/apache/hu …

在编译hudi master的过程中，遇到了编译错误，提示unable to find valid certification path to requested target 错误，无法下载依赖的jar与pom文件 使用命令行curl可以下载文件，通过谷歌搜索查到是java访问https网址证书问题 解决方法 下载需要访问的网址证书，因为hudi源码中的pom.xml文件中默认使用的是https://repo.maven.apache.org/maven2/,很多的依赖无法下载，我修改为https://packages.confluent.io/maven/ ，所以先下载packages.confluent.io的证书 点击浏览器中的锁图标 然后选择证书，点击详细信息，选择复制到文件

The field of data science relies heavily on the predictive capability of Machine Learning (ML) algorithms. Python offers an opportune playground for experimenting with these algorithms due to the readability and syntactical efficiency of the language. …

Capture all row-based DML changes from the Oracle database and stream these changes to Kafka. The change data capture logic is based on the Oracle LogMiner solution. Test environment preparation (1)A linux cloud server is configured as a 2 core 4G 2M bandwidth 50G system disk (2)install docker on cloud…

In Flink, the Transformation operator is to convert one or more DataStreams into a new DataStream, which can combine multiple transformations into a complex data stream topology. …

Spark operators can be roughly divided into the following three categories: 1. Transformation data operator of the Value data type, this transformation does not trigger the submission of the job, the data item for the processing is Value type data. 2. The Transfromation operator of the Key-Value data type. This…

One of the benefits of Zeppelin is that you can switch easily between different code,we can use datepicker to build a Zeppelin dynamic form in its own paragraph. The Front-end Angular API is then used to bind the selected date range to paragraphs containing jdbc SQL with a date predicate …

In our work, we often encounter the problem of doing various aggregation calculations based on time or other dimensions. Problem 1 : user login to a website ,statistical activity duration Suppose there is a DataFrame of user logins to a website, for instance: +----------------+----------+ | user_name|login_date| +----------------+----------+ |smith |2020-01-04| |bob |2020-01-04| |bob |2020-01-06| |john |2020-01-10| |smith |2020-01-11| +----------------+----------+

Eureka registry/service discovery framework Eureka is a service discovery framework developed by Netflix, itself a REST-based service that is used to locate middle-tier services running in AWS domains for load balancing and middle-tier service failover purposes. springCloud integrates it into its subproject spring-cloud-netflix to implement the SpringCloud’s Service Discovery feature. …