GoogleCloudDataproc/spark-bigquery-connector issues and pull requests

#1299 - [Glue 4.0 integration]

Issue - State: open - Opened by Repomano 3 days ago

#1298 - What determines the Query API vs. Storage API to be used?

Issue - State: open - Opened by whittid4 3 days ago

#1297 - Support for spark version 3.5.0 - support required for UNTYPED_SCALA_UDF version

Issue - State: open - Opened by VIKCT001 5 days ago - 2 comments

#1296 - Setting a different client api endpoiint for Bigquery

Issue - State: open - Opened by EmilioIppoliti 6 days ago - 1 comment

#1295 - org.apache.spark.sql.AnalysisException: [UNSUPPORTED_DESERIALIZER.DATA_TYPE_MISMATCH] The deserializer is not supported: need a(n) "ARRAY" field but got "MAP<STRING, STRING>".

Issue - State: open - Opened by VIKCT001 9 days ago - 5 comments

#1294 - exception : BigQueryException: Provided Schema does not match Table

Issue - State: closed - Opened by VIKCT001 11 days ago - 4 comments

#1293 - Issue #1290: Stopped using metadata for optimized count path

Pull Request - State: open - Opened by vishalkarve15 14 days ago - 5 comments

#1292 - BigQuery RANGE (and INTERVAL) support

Issue - State: open - Opened by 0rnella 17 days ago - 1 comment
Labels: enhancement

#1291 - Error during reading numeric & bignumeric

Issue - State: closed - Opened by flo076 17 days ago - 2 comments

#1290 - Issue with BigQuery counts in Spark after writing to table

Issue - State: open - Opened by rchynoweth 24 days ago - 2 comments

#1289 - Update CHANGES.md

Pull Request - State: closed - Opened by isha97 24 days ago

#1288 - Change log to debug for exceptions while fetching Identity token

Pull Request - State: closed - Opened by isha97 24 days ago

#1287 - Upgrade Dependencies

Pull Request - State: closed - Opened by isha97 24 days ago - 2 comments

#1286 - df.write()not working with Spark-Java code on serverless

Issue - State: closed - Opened by saikrishnab27 25 days ago - 1 comment

#1285 - Unable to write to BigQuery table with require_partition_filter = true when spark.sql.sources.partitionOverwriteMode is set to DYNAMIC

Issue - State: open - Opened by ktchana 26 days ago - 1 comment

#1284 - Making BigQueryClientFactory Kryo serializable

Pull Request - State: open - Opened by davidrabinowitz about 1 month ago - 10 comments

#1283 - Fix readme for configuring bigNumeric defaults

Pull Request - State: closed - Opened by isha97 about 1 month ago

#1282 - Issue #1175: Add details to schema mismatch message

Pull Request - State: closed - Opened by vishalkarve15 about 1 month ago - 6 comments

#1281 - Configure alternative BigNumeric precision and scale defaults

Pull Request - State: closed - Opened by isha97 about 1 month ago - 5 comments

#1280 - Spark structured streaming jobs failing with Bigquery session expired

Issue - State: closed - Opened by caspherola about 1 month ago - 3 comments

#1279 - Add support for BigQuery tables with Collation

Issue - State: closed - Opened by jster1357 about 1 month ago - 2 comments

#1278 - Issue with Map Type support (pyspark)

Issue - State: closed - Opened by violetautumn about 2 months ago - 1 comment

#1277 - `bigquery.tables.create` Permission required on appending data to a BigQuery Table

Issue - State: open - Opened by smic-datalabs-jdcastro about 2 months ago - 2 comments

#1276 - Upgrade Dependencies

Pull Request - State: closed - Opened by isha97 about 2 months ago - 2 comments

#1275 - Writing spark data with timestamp_ntz/timestamp to bigquery column datetime

Issue - State: closed - Opened by ne250040 about 2 months ago - 1 comment

#1274 - cloud cost when using spark bq connector

Issue - State: open - Opened by kaiseu about 2 months ago

#1273 - Update CHANGES.md for next release

Pull Request - State: closed - Opened by isha97 2 months ago

#1272 - Move misplaced entry in CHANGES file

Pull Request - State: open - Opened by rafalh 2 months ago - 1 comment

#1271 - Add to changes.md

Pull Request - State: closed - Opened by vishalkarve15 2 months ago - 2 comments

#1270 - Issue #1263 : Support ClusteredFields in Direct Write

Pull Request - State: closed - Opened by isha97 2 months ago - 3 comments

#1269 - Update spark-bigquery-support version

Pull Request - State: closed - Opened by MattiaIppoliti 2 months ago - 1 comment

#1268 - Empty array filtering issue

Issue - State: open - Opened by vokuk-walkme 2 months ago - 1 comment

#1267 - Implement OpenLineage spark-extension-interfaces

Pull Request - State: closed - Opened by ddebowczyk92 2 months ago - 8 comments

#1266 - Reading a view results in a empty data frame

Issue - State: open - Opened by tom-s-powell 2 months ago - 7 comments

#1265 - Add additional parentheses for EqualNullSafe filter generation

Pull Request - State: closed - Opened by tom-s-powell 2 months ago - 2 comments

#1264 - Failed to write into bigquery using spark 3.2.1

Issue - State: closed - Opened by krishna00000785 2 months ago - 2 comments

#1263 - "ClusteredFields" creation not supported in Direct write method from Pyspark to BigQuery

Issue - State: closed - Opened by deena10545 2 months ago - 2 comments

#1262 - Big Query Full Table Read - Some Partitions take longer bqScanTime as compared to other partitions with similar number of records

Issue - State: closed - Opened by josepaul1988 2 months ago - 4 comments

#1261 - Adding IdentityToken header in readRows call

Pull Request - State: closed - Opened by isha97 3 months ago - 4 comments

#1260 - proxyAddress variable is not used if default application credentials method is used

Issue - State: closed - Opened by milicevica23 3 months ago - 4 comments

#1259 - Encode snapshotTimeMillis in view materialization query

Pull Request - State: closed - Opened by tom-s-powell 3 months ago - 2 comments

#1258 - Issue #1254: fix getting partitioning fields for pseudo columns

Pull Request - State: closed - Opened by vishalkarve15 3 months ago - 5 comments

#1257 - Unable to write decimal values with scale > 9 to bigquery using dataproc and spark bigquery connector.

Issue - State: closed - Opened by abhilash499 3 months ago - 2 comments

#1256 - NullPointerException - Error creating destination table using the following query

Issue - State: closed - Opened by srikanthnagidi 3 months ago - 2 comments

#1255 - LoadingCache for identity token supplier

Pull Request - State: closed - Opened by isha97 3 months ago

#1254 - NullPointerException when using Java connector with time partitioned tables

Issue - State: closed - Opened by rafalh 3 months ago - 3 comments

#1253 - Make BigQueryClientFactory Kryo serializable.

Pull Request - State: closed - Opened by davidrabinowitz 3 months ago - 2 comments

#1252 - Viewing connector metrics

Issue - State: closed - Opened by Swebask 3 months ago - 4 comments

#1251 - Add equals and hashcode in IdentityTokenSupplier

Pull Request - State: closed - Opened by isha97 3 months ago - 2 comments

#1250 - Issue #1043: fixed indirect write overwrite dropping policy tags

Pull Request - State: closed - Opened by vishalkarve15 3 months ago - 3 comments

#1249 - Issue #1244, Issue #1043 Use existing field properties and only update as per ALLOW_FIELD_RELAXATION

Pull Request - State: closed - Opened by isha97 3 months ago - 10 comments

#1248 - Update Changes for 0.39.1 release

Pull Request - State: closed - Opened by isha97 3 months ago

#1247 - Disable internal-ip-only for acceptance test clusters

Pull Request - State: closed - Opened by isha97 3 months ago - 1 comment

#1246 - Permission bigquery.tables.create required on already materialised view

Issue - State: closed - Opened by k-charalampous 4 months ago - 1 comment

#1245 - Use SessionState to obtain Hadoop configuration

Pull Request - State: open - Opened by tom-s-powell 4 months ago

#1244 - Disable Schema Changes

Issue - State: closed - Opened by gmiano 4 months ago - 2 comments

#1243 - Issue #1126: Making SparkBigQueryConfig Kryo compatible

Pull Request - State: closed - Opened by davidrabinowitz 4 months ago - 2 comments

#1242 - Spark BQ connector unable to handle timestamps - 0001-01-01 00:00:00

Issue - State: closed - Opened by Murli16 4 months ago - 3 comments

#1241 - Shading updates for release 0.39.1

Pull Request - State: closed - Opened by davidrabinowitz 4 months ago - 1 comment

#1240 - Issue #1223: gRPC status creates 'object not serializable' errors

Pull Request - State: closed - Opened by davidrabinowitz 4 months ago - 1 comment

#1239 - Allow GCS bucket to be supplied including a scheme

Pull Request - State: closed - Opened by tom-s-powell 4 months ago - 1 comment

#1238 - Fix the log message regarding number of partitions received from read…

Pull Request - State: closed - Opened by isha97 4 months ago

#1237 - BigQuery Storage API always returning 200 partitions

Issue - State: closed - Opened by rcanzanese 4 months ago - 1 comment

#1236 - Adding shading verification

Pull Request - State: closed - Opened by davidrabinowitz 4 months ago - 5 comments

#1235 - Relocate net.bytebuddy:bytebuddy dependency

Pull Request - State: closed - Opened by tom-s-powell 4 months ago - 1 comment

#1234 - BigNumeric precision is too wide (76), Spark can only handle decimal types with max precision of 38

Issue - State: closed - Opened by John15321 4 months ago - 6 comments

#1233 - Configuring DataProc Cluster to Publish BigQuery Metrics on GCP

Issue - State: open - Opened by gmiano 4 months ago - 1 comment
Labels: enhancement

#1232 - Indirect write to existing datetime column not possible

Issue - State: closed - Opened by cheare 4 months ago - 2 comments

#1231 - Upgrade dependencies

Pull Request - State: closed - Opened by isha97 4 months ago - 2 comments

#1230 - Updating the Changes file

Pull Request - State: closed - Opened by davidrabinowitz 4 months ago - 1 comment

#1229 - TIMESTAMP appears to add an hour compared to DATETIME

Issue - State: closed - Opened by Naboo22 4 months ago - 2 comments

#1228 - Unable to write to a partitioned table in spark 3.5 with connector version spark-3.4-bigquery-0.34.0

Issue - State: closed - Opened by hierr 5 months ago - 6 comments

#1227 - Optimized MERGE query for dynamic partition overwrite in Time partitioned table

Pull Request - State: closed - Opened by isha97 5 months ago - 2 comments

#1226 - PySpark virtualenv: Missing dependencies lead to No FileSystem for scheme "gs"

Issue - State: closed - Opened by GuiBou165 5 months ago - 3 comments

#1225 - Fix multi-release jar shading

Pull Request - State: closed - Opened by vishalkarve15 5 months ago - 3 comments

#1224 - Request to Stop Enforcing Delete Permissions on Materialization Dataset

Issue - State: closed - Opened by reynoldspravindev 5 months ago - 1 comment

#1223 - error while writing with many timestamp data

Issue - State: closed - Opened by iqbalaey 5 months ago

#1222 - add option to request lz4 compressed ReadRowsResponse

Pull Request - State: closed - Opened by yqiu2 5 months ago - 5 comments

#1221 - Add config option for snapshot time

Pull Request - State: closed - Opened by tom-s-powell 5 months ago - 1 comment

#1220 - Upgrade dependencies

Pull Request - State: closed - Opened by isha97 5 months ago - 6 comments

#1219 - Remove spark 2.4 connector from the release

Pull Request - State: closed - Opened by isha97 5 months ago - 3 comments

#1218 - Publishing write session metrics from connector

Pull Request - State: closed - Opened by isha97 5 months ago - 5 comments

#1217 - Update Changes for next release

Pull Request - State: closed - Opened by isha97 5 months ago

#1216 - Support predicate pushdown for DATETIME

Pull Request - State: closed - Opened by isha97 5 months ago - 1 comment

#1215 - Predicate pushdown doesn't work with DateTime BQ field - Spark 3.5, connector version 0.37

Issue - State: closed - Opened by yunus 5 months ago - 1 comment

#1214 - Error: This connector was made for Scala null, it was not meant to run on Scala 2.12

Issue - State: closed - Opened by malhomaid 5 months ago - 2 comments

#1213 - Add `createTable(TableInfo)` method to `BigQueryClient`

Issue - State: open - Opened by jphalip 6 months ago
Labels: enhancement

#1212 - Support OpenLineage in spark-3.x-bigquery connectors

Pull Request - State: closed - Opened by codelixir 6 months ago - 10 comments

#1211 - Sending Read Metrics and metadata to Spark listener

Pull Request - State: closed - Opened by isha97 6 months ago - 1 comment

#1210 - How to handle column datatype change?

Issue - State: closed - Opened by nandinir-db 6 months ago - 9 comments

#1209 - java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.encoders.ExpressionEncoder org.apache.spark.sql.catalyst.encoders.RowEncoder.apply(org.apache.spark.sql.types.StructType)'

Issue - State: open - Opened by CatalinMihaiIonescu 6 months ago - 6 comments

#1208 - BIGNUMERIC Precision Handling: Inaccurate Decimal Values

Issue - State: closed - Opened by nabeelq 6 months ago - 2 comments

#1207 - BigQuery Pushdown filtering on Spark 3.4.2

Issue - State: open - Opened by sid-habu 6 months ago - 11 comments
Labels: enhancement

GitHub / GoogleCloudDataproc/spark-bigquery-connector issues and pull requests