[FLINK-35543][HIVE] Upgrade Hive 2.3 connector to version 2.3.10 (#24905)

pull/24938/head
Cheng Pan 7 months ago committed by GitHub
parent 93b9bdf798
commit 6c05981948
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

@ -54,6 +54,7 @@ Flink 支持以下的 Hive 版本。
- 2.3.7
- 2.3.8
- 2.3.9
- 2.3.10
- 3.1
- 3.1.0
- 3.1.1
@ -87,10 +88,10 @@ export HADOOP_CLASSPATH=`hadoop classpath`
下表列出了所有可用的 Hive jar。您可以选择一个并放在 Flink 发行版的`/lib/` 目录中。
| Metastore version | Maven dependency | SQL Client JAR |
|:------------------|:---------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.3.0 - 2.3.9 | `flink-sql-connector-hive-2.3.9` | {{< stable >}}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.9{{< scala_version >}}/{{< version >}}/flink-sql-connector-hive-2.3.9{{< scala_version >}}-{{< version >}}.jar) {{< /stable >}}{{< unstable >}} Only available for stable releases {{< /unstable >}} |
| 3.0.0 - 3.1.3 | `flink-sql-connector-hive-3.1.3` | {{< stable >}}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-3.1.3{{< scala_version >}}/{{< version >}}/flink-sql-connector-hive-3.1.3{{< scala_version >}}-{{< version >}}.jar) {{< /stable >}}{{< unstable >}} Only available for stable releases {{< /unstable >}} |
| Metastore version | Maven dependency | SQL Client JAR |
|:------------------|:----------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.3.0 - 2.3.10 | `flink-sql-connector-hive-2.3.10` | {{< stable >}}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.10{{< scala_version >}}/{{< version >}}/flink-sql-connector-hive-2.3.10{{< scala_version >}}-{{< version >}}.jar) {{< /stable >}}{{< unstable >}} Only available for stable releases {{< /unstable >}} |
| 3.0.0 - 3.1.3 | `flink-sql-connector-hive-3.1.3` | {{< stable >}}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-3.1.3{{< scala_version >}}/{{< version >}}/flink-sql-connector-hive-3.1.3{{< scala_version >}}-{{< version >}}.jar) {{< /stable >}}{{< unstable >}} Only available for stable releases {{< /unstable >}} |
#### 用户定义的依赖项

@ -54,6 +54,7 @@ Flink supports the following Hive versions.
- 2.3.7
- 2.3.8
- 2.3.9
- 2.3.10
- 3.1
- 3.1.0
- 3.1.1
@ -89,10 +90,10 @@ There are two ways to add Hive dependencies. First is to use Flink's bundled Hiv
The following tables list all available bundled hive jars. You can pick one to the `/lib/` directory in Flink distribution.
| Metastore version | Maven dependency | SQL Client JAR |
|:------------------|:---------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.3.0 - 2.3.9 | `flink-sql-connector-hive-2.3.9` | {{< stable >}}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.9{{< scala_version >}}/{{< version >}}/flink-sql-connector-hive-2.3.9{{< scala_version >}}-{{< version >}}.jar) {{< /stable >}}{{< unstable >}} Only available for stable releases {{< /unstable >}} |
| 3.0.0 - 3.1.3 | `flink-sql-connector-hive-3.1.3` | {{< stable >}}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-3.1.3{{< scala_version >}}/{{< version >}}/flink-sql-connector-hive-3.1.3{{< scala_version >}}-{{< version >}}.jar) {{< /stable >}}{{< unstable >}} Only available for stable releases {{< /unstable >}} |
| Metastore version | Maven dependency | SQL Client JAR |
|:------------------|:----------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.3.0 - 2.3.10 | `flink-sql-connector-hive-2.3.10` | {{< stable >}}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.10{{< scala_version >}}/{{< version >}}/flink-sql-connector-hive-2.3.10{{< scala_version >}}-{{< version >}}.jar) {{< /stable >}}{{< unstable >}} Only available for stable releases {{< /unstable >}} |
| 3.0.0 - 3.1.3 | `flink-sql-connector-hive-3.1.3` | {{< stable >}}[Download](https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-3.1.3{{< scala_version >}}/{{< version >}}/flink-sql-connector-hive-3.1.3{{< scala_version >}}-{{< version >}}.jar) {{< /stable >}}{{< unstable >}} Only available for stable releases {{< /unstable >}} |
#### User defined dependencies

@ -40,6 +40,7 @@ public class HiveShimLoader {
public static final String HIVE_VERSION_V2_3_7 = "2.3.7";
public static final String HIVE_VERSION_V2_3_8 = "2.3.8";
public static final String HIVE_VERSION_V2_3_9 = "2.3.9";
public static final String HIVE_VERSION_V2_3_10 = "2.3.10";
public static final String HIVE_VERSION_V3_1_0 = "3.1.0";
public static final String HIVE_VERSION_V3_1_1 = "3.1.1";
public static final String HIVE_VERSION_V3_1_2 = "3.1.2";
@ -85,6 +86,9 @@ public class HiveShimLoader {
if (v.startsWith(HIVE_VERSION_V2_3_9)) {
return new HiveShimV239();
}
if (v.startsWith(HIVE_VERSION_V2_3_10)) {
return new HiveShimV2310();
}
if (v.startsWith(HIVE_VERSION_V3_1_0)) {
return new HiveShimV310();
}

@ -0,0 +1,22 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.flink.table.catalog.hive.client;
/** Shim for Hive version 2.3.10. */
public class HiveShimV2310 extends HiveShimV239 {}

@ -116,7 +116,6 @@ import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import static org.apache.flink.configuration.ExecutionOptions.RUNTIME_MODE;
import static org.apache.flink.table.api.config.TableConfigOptions.TABLE_DML_SYNC;
@ -869,10 +868,6 @@ public class HiveServer2Endpoint implements TCLIService.Iface, SqlGatewayEndpoin
.inputProtocolFactory(
new TBinaryProtocol.Factory(
true, true, maxMessageSize, maxMessageSize))
.requestTimeout(requestTimeoutMs)
.requestTimeoutUnit(TimeUnit.MILLISECONDS)
.beBackoffSlotLength(backOffSlotLengthMs)
.beBackoffSlotLengthUnit(TimeUnit.MILLISECONDS)
.executorService(executor));
} catch (Exception e) {
throw new SqlGatewayException("Failed to build the server.", e);

@ -46,6 +46,7 @@ public class HiveRunnerShimLoader {
case HiveShimLoader.HIVE_VERSION_V2_3_7:
case HiveShimLoader.HIVE_VERSION_V2_3_8:
case HiveShimLoader.HIVE_VERSION_V2_3_9:
case HiveShimLoader.HIVE_VERSION_V2_3_10:
case HiveShimLoader.HIVE_VERSION_V3_1_0:
case HiveShimLoader.HIVE_VERSION_V3_1_1:
case HiveShimLoader.HIVE_VERSION_V3_1_2:

@ -29,8 +29,8 @@ under the License.
<version>1.20-SNAPSHOT</version>
</parent>
<artifactId>flink-sql-connector-hive-2.3.9_${scala.binary.version}</artifactId>
<name>Flink : Connectors : SQL : Hive 2.3.9</name>
<artifactId>flink-sql-connector-hive-2.3.10_${scala.binary.version}</artifactId>
<name>Flink : Connectors : SQL : Hive 2.3.10</name>
<packaging>jar</packaging>
@ -49,7 +49,7 @@ under the License.
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>2.3.9</version>
<version>2.3.10</version>
<optional>${flink.markBundledAsOptional}</optional>
<exclusions>
<exclusion>
@ -60,10 +60,6 @@ under the License.
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>org.pentaho</groupId>
<artifactId>pentaho-aggdesigner-algorithm</artifactId>
</exclusion>
<!-- Rely on the Guava version of Hadoop. -->
<exclusion>
<groupId>com.google.guava</groupId>

@ -1,4 +1,4 @@
flink-sql-connector-hive-2.3.9
flink-sql-connector-hive-2.3.10
Copyright 2014-2024 The Apache Software Foundation
This product includes software developed at
@ -6,7 +6,7 @@ The Apache Software Foundation (http://www.apache.org/).
This project bundles the following dependencies under the Apache Software License 2.0. (http://www.apache.org/licenses/LICENSE-2.0.txt)
- org.apache.hive:hive-exec:2.3.9
- org.apache.hive:hive-exec:2.3.10
This project bundles the following dependencies under the BSD license.
See bundled license files for details.
@ -16,13 +16,12 @@ See bundled license files for details.
The bundled Apache Hive org.apache.hive:hive-exec dependency bundles the following dependencies under
the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
- com.fasterxml.jackson.core:jackson-annotations:2.6.3
- com.fasterxml.jackson.core:jackson-core:2.6.3
- com.fasterxml.jackson.core:jackson-databind:2.6.3
- com.fasterxml.jackson.core:jackson-annotations:2.12.0
- com.fasterxml.jackson.core:jackson-core:2.12.0
- com.fasterxml.jackson.core:jackson-databind:2.12.0
- com.google.guava:guava:14.0.1
- com.googlecode.javaewah:JavaEWAH:0.3.2
- com.tdunning:json:1.8
- commons-lang:commons-lang:2.6
- io.airlift:aircompressor:0.8
- javax.jdo:jdo-api:3.0.1
- joda-time:joda-time:2.8.1
@ -34,28 +33,22 @@ the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
- org.apache.calcite:calcite-druid:1.10.0
- org.apache.calcite:calcite-linq4j:1.10.0
- org.apache.calcite.avatica:avatica:1.8.0
- org.apache.commons:commons-lang:2.6
- org.apache.commons:commons-lang3:3.1
- org.apache.hive.shims:hive-shims-0.23:2.3.9
- org.apache.hive.shims:hive-shims-common:2.3.9
- org.apache.hive:hive-common:2.3.9
- org.apache.hive:hive-llap-client:2.3.9
- org.apache.hive:hive-llap-common:2.3.9
- org.apache.hive:hive-metastore:2.3.9
- org.apache.hive:hive-serde:2.3.9
- org.apache.hive:hive-service-rpc:2.3.9
- org.apache.hive.shims:hive-shims-0.23:2.3.10
- org.apache.hive.shims:hive-shims-common:2.3.10
- org.apache.hive:hive-common:2.3.10
- org.apache.hive:hive-llap-client:2.3.10
- org.apache.hive:hive-llap-common:2.3.10
- org.apache.hive:hive-metastore:2.3.10
- org.apache.hive:hive-serde:2.3.10
- org.apache.hive:hive-service-rpc:2.3.10
- org.apache.hive:hive-storage-api:2.4.0
- org.apache.hive:spark-client:2.3.9
- org.apache.hive:spark-client:2.3.10
- org.apache.orc:orc-core:1.3.4
- org.apache.orc:orc-tools:1.3.4
- org.apache.parquet:parquet-hadoop-bundle:1.8.1
- org.apache.thrift:libthrift:0.7.0
- org.apache.thrift:libthrift:0.9.3
- org.apache.thrift:libthrift:0.14.1
- org.apache.thrift:libfb303:0.9.3
- org.codehaus.jackson:jackson-core-asl:1.9.11
- org.codehaus.jackson:jackson-core-asl:1.9.13
- org.codehaus.jackson:jackson-mapper-asl:1.9.11
- org.codehaus.jackson:jackson-mapper-asl:1.9.13
- org.objenesis:objenesis:2.1
The bundled Apache Hive org.apache.hive:hive-exec dependency bundles the following dependencies under the BSD license.
@ -66,7 +59,6 @@ See bundled license files for details.
- com.esotericsoftware:reflectasm:1.10.1
- com.google.protobuf:protobuf-java:2.5.0
- javolution:javolution:5.5.1
- org.jodd:jodd-core:3.5.2
The bundled Apache Hive org.apache.hive:hive-exec dependency bundles the following dependencies under the MIT/X11 license.
See bundled license files for details.

@ -1,25 +0,0 @@
/*
* Copyright (c) 2003-2020, Jodd Team All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
* LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
* NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

@ -78,7 +78,7 @@ under the License.
</property>
</activation>
<modules>
<module>flink-sql-connector-hive-2.3.9</module>
<module>flink-sql-connector-hive-2.3.10</module>
<module>flink-sql-connector-hive-3.1.3</module>
</modules>
</profile>

@ -175,7 +175,7 @@ under the License.
to revisit the impact at that time.
-->
<minikdc.version>3.2.4</minikdc.version>
<hive.version>2.3.9</hive.version>
<hive.version>2.3.10</hive.version>
<orc.version>1.5.6</orc.version>
<japicmp.referenceVersion>1.19.0</japicmp.referenceVersion>
<japicmp.outputDir>tools/japicmp-output</japicmp.outputDir>

@ -19,6 +19,6 @@
# This file lists modules which define additional dependencies, not shown by the maven shade plugin output.
# These are usually undeclared shaded (or otherwise included) dependencies from transitive dependencies.
flink-sql-connector-hive-2.3.9
flink-sql-connector-hive-2.3.10
flink-sql-connector-hive-3.1.3
flink-python

Loading…
Cancel
Save