一、环境准备 hadoop 2.10.1 hive 2.3.6 hudi 0.9 flink 1.12.2 各组件的安装配置就不介绍了，可以google搜索下就能找到 二、flink+hudi实战步骤 flink 单机或者集群配置好，下载hudi依赖包 wget https://repo.maven.apache.org/maven2/org/apache/hu …

The field of data science relies heavily on the predictive capability of Machine Learning (ML) algorithms. Python offers an opportune playground for experimenting with these algorithms due to the readability and syntactical efficiency of the language. …

Capture all row-based DML changes from the Oracle database and stream these changes to Kafka. The change data capture logic is based on the Oracle LogMiner solution. Test environment preparation (1)A linux cloud server is configured as a 2 core 4G 2M bandwidth 50G system disk (2)install docker on cloud…

In Flink, the Transformation operator is to convert one or more DataStreams into a new DataStream, which can combine multiple transformations into a complex data stream topology. …

Spark operators can be roughly divided into the following three categories: 1. Transformation data operator of the Value data type, this transformation does not trigger the submission of the job, the data item for the processing is Value type data. 2. The Transfromation operator of the Key-Value data type. This…

One of the benefits of Zeppelin is that you can switch easily between different code,we can use datepicker to build a Zeppelin dynamic form in its own paragraph. The Front-end Angular API is then used to bind the selected date range to paragraphs containing jdbc SQL with a date predicate …

In our work, we often encounter the problem of doing various aggregation calculations based on time or other dimensions. Problem 1 : user login to a website ,statistical activity duration Suppose there is a DataFrame of user logins to a website, for instance: +----------------+----------+ | user_name|login_date| +----------------+----------+ |smith |2020-01-04| |bob |2020-01-04| |bob |2020-01-06| |john |2020-01-10| |smith |2020-01-11| +----------------+----------+

Eureka registry/service discovery framework Eureka is a service discovery framework developed by Netflix, itself a REST-based service that is used to locate middle-tier services running in AWS domains for load balancing and middle-tier service failover purposes. springCloud integrates it into its subproject spring-cloud-netflix to implement the SpringCloud’s Service Discovery feature. …