Mirrors
/
flink
mirror of https://github.com/apache/flink.git
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[FLINK-32511][FileSystems] Upgrade GCS connector to 2.2.15. This closes #22930

Browse Source 
* [FLINK-32511] Upgrade GCS connector to 2.2.15

* Update NOTICE file
pull/21544/merge
Jayadeep Jayaraman 2 years ago committed by GitHub
parent 4b61e4b79e
commit f6d09c5c01
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 53 additions and 25 deletions
Show Stats Download Patch File Download Diff File

30
.idea/vcs.xml
Unescape Escape View File

@ -15,6 +15,34 @@
<option name="issueRegexp" value="#(\d+)" />
<option name="linkRegexp" value="https://github.com/apache/flink/pull/$1" />
</IssueNavigationLink>
<IssueNavigationLink>
<option name="issueRegexp" value="\bb/(\d+)(#\w+)?\b" />
<option name="linkRegexp" value="https://buganizer.corp.google.com/issues/$1$2" />
</IssueNavigationLink>
<IssueNavigationLink>
<option name="issueRegexp" value="\b(?:BUG=|FIXED=)(\d+)\b" />
<option name="linkRegexp" value="https://buganizer.corp.google.com/issues/$1" />
</IssueNavigationLink>
<IssueNavigationLink>
<option name="issueRegexp" value="\b(?:cl/|cr/|OCL=|DIFFBASE=|ROLLBACK_OF=)(\d+)\b" />
<option name="linkRegexp" value="https://critique.corp.google.com/$1" />
</IssueNavigationLink>
<IssueNavigationLink>
<option name="issueRegexp" value="\bomg/(\d+)\b" />
<option name="linkRegexp" value="https://omg.corp.google.com/$1" />
</IssueNavigationLink>
<IssueNavigationLink>
<option name="issueRegexp" value="\b(?:go/|goto/)([^,.&lt;&gt;()&quot;\s]+(?:[.,][^,.&lt;&gt;()&quot;\s]+)*)" />
<option name="linkRegexp" value="https://goto.google.com/$1" />
</IssueNavigationLink>
<IssueNavigationLink>
<option name="issueRegexp" value="\bcs/([^\s]+[\w$])" />
<option name="linkRegexp" value="https://cs.corp.google.com/search/?q=$1" />
</IssueNavigationLink>
<IssueNavigationLink>
<option name="issueRegexp" value="(LINT\.IfChange)|(LINT\.ThenChange)" />
<option name="linkRegexp" value="https://goto.google.com/ifthisthenthatlint" />
</IssueNavigationLink>
</list>
</option>
</component>
@ -22,4 +50,4 @@
<mapping directory="$PROJECT_DIR$" vcs="Git" />
<mapping directory="$PROJECT_DIR$/docs/themes/book" vcs="Git" />
</component>
</project>
</project>

4
docs/content.zh/docs/deployment/filesystems/gcs.md
Unescape Escape View File

@ -55,7 +55,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place
Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it.
`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.11) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.15.0) library to provide `RecoverableWriter` support.
`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.15) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.15.0) library to provide `RecoverableWriter` support.
This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}).
@ -68,7 +68,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/
### Configuration
The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.11/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.15/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in `flink-conf.yaml`. Flink will internally translate this back to `fs.gs.http.connect-timeout`.

4
docs/content/docs/deployment/filesystems/gcs.md
Unescape Escape View File

@ -55,7 +55,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place
Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it.
`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.11) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.15.0) library to provide `RecoverableWriter` support.
`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.15) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.15.0) library to provide `RecoverableWriter` support.
This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}).
@ -68,7 +68,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/
### Configuration
The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.11/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.15/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in `flink-conf.yaml`. Flink will internally translate this back to `fs.gs.http.connect-timeout`.

4
flink-filesystems/flink-gs-fs-hadoop/pom.xml
Unescape Escape View File

@ -35,9 +35,9 @@ under the License.
<!-- If updating these dependency versions, please also update the corresponding links -->
<!-- in the GCS file system documentation. -->
<fs.gs.sdk.version>2.15.0</fs.gs.sdk.version>
<fs.gs.connector.version>hadoop3-2.2.11</fs.gs.connector.version>
<fs.gs.connector.version>hadoop3-2.2.15</fs.gs.connector.version>
<!-- Set this to the highest version of grpc artifacts from gcs-connector and google-cloud-storage -->
<fs.gs.grpc.version>1.50.2</fs.gs.grpc.version>
<fs.gs.grpc.version>1.50.3</fs.gs.grpc.version>
</properties>
<dependencies>

36
flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE
Unescape Escape View File

@ -20,10 +20,10 @@ This project bundles the following dependencies under the Apache Software Licens
- com.google.apis:google-api-services-iamcredentials:v1-rev20211203-2.0.0
- com.google.apis:google-api-services-storage:v1-rev20220705-2.0.0
- com.google.auto.value:auto-value-annotations:1.10
- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.11
- com.google.cloud.bigdataoss:gcsio:2.2.11
- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.11
- com.google.cloud.bigdataoss:util:2.2.11
- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.15
- com.google.cloud.bigdataoss:gcsio:2.2.15
- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.15
- com.google.cloud.bigdataoss:util:2.2.15
- com.google.cloud:google-cloud-core-grpc:2.8.27
- com.google.cloud:google-cloud-core-http:2.8.27
- com.google.cloud:google-cloud-core:2.8.27
@ -42,20 +42,20 @@ This project bundles the following dependencies under the Apache Software Licens
- com.google.oauth-client:google-oauth-client:1.34.1
- com.lmax:disruptor:3.4.2
- commons-codec:commons-codec:1.15
- io.grpc:grpc-alts:1.50.2
- io.grpc:grpc-api:1.50.2
- io.grpc:grpc-auth:1.50.2
- io.grpc:grpc-census:1.50.2
- io.grpc:grpc-context:1.50.2
- io.grpc:grpc-core:1.50.2
- io.grpc:grpc-googleapis:1.50.2
- io.grpc:grpc-grpclb:1.50.2
- io.grpc:grpc-netty-shaded:1.50.2
- io.grpc:grpc-protobuf-lite:1.50.2
- io.grpc:grpc-protobuf:1.50.2
- io.grpc:grpc-services:1.50.2
- io.grpc:grpc-stub:1.50.2
- io.grpc:grpc-xds:1.50.2
- io.grpc:grpc-alts:1.50.3
- io.grpc:grpc-api:1.50.3
- io.grpc:grpc-auth:1.50.3
- io.grpc:grpc-census:1.50.3
- io.grpc:grpc-context:1.50.3
- io.grpc:grpc-core:1.50.3
- io.grpc:grpc-googleapis:1.50.3
- io.grpc:grpc-grpclb:1.50.3
- io.grpc:grpc-netty-shaded:1.50.3
- io.grpc:grpc-protobuf-lite:1.50.3
- io.grpc:grpc-protobuf:1.50.3
- io.grpc:grpc-services:1.50.3
- io.grpc:grpc-stub:1.50.3
- io.grpc:grpc-xds:1.50.3
- io.opencensus:opencensus-api:0.31.1
- io.opencensus:opencensus-contrib-exemplar-util:0.31.0
- io.opencensus:opencensus-contrib-grpc-metrics:0.31.0

Write Preview
Loading…
Cancel
Save