apache/incubator-celeborn issues and pull requests

#2290 - [CELEBORN-1266] Improve log of current failed workers for WorkerStatusTracker

Pull Request - State: open - Opened by SteNicholas 8 months ago

#2289 - [CELEBORN-1265] Fix batches read metric for gluten columnar shuffle

Pull Request - State: open - Opened by kerwin-zk 8 months ago - 2 comments

#2288 - [CELEBORN-1252][FOLLOWUP] Fix Worker#computeResourceConsumption NullPointerException for userResourceConsumption that does not contain given userIdentifier

Pull Request - State: closed - Opened by SteNicholas 8 months ago - 1 comment

#2287 - [CELEBORN-1174][FOLLOWUP][0.4] Fix Worker#computeResourceConsumption NullPointerException with null subResourceConsumptions

Pull Request - State: closed - Opened by SteNicholas 8 months ago

#2286 - [CELEBORN-1174][FOLLOWUP] Fix Worker#computeResourceConsumption NullPointerException with null subResourceConsumptions

Pull Request - State: closed - Opened by SteNicholas 8 months ago - 1 comment

#2285 - [CELEBORN-1264][IMPROVEMENT] FsConfigServiceImpl support user level config

Pull Request - State: open - Opened by AngersZhuuuu 8 months ago - 5 comments

#2284 - [CELEBORN-1262] Bump Spark from 3.3.3 to 3.3.4

Pull Request - State: closed - Opened by SteNicholas 8 months ago - 2 comments

#2283 - [CELEBORN-1263] Fix Master HA mode without internal port error

Pull Request - State: closed - Opened by wxplovecc 8 months ago - 2 comments

#2282 - [CELEBORN-1249] Add LICENSE of Celeborn Web

Pull Request - State: open - Opened by labbomb 8 months ago - 2 comments

#2281 - [CELEBORN-1257] Adds a secured port in Celeborn Master for secure communication with LifecycleManager

Pull Request - State: closed - Opened by otterc 8 months ago - 1 comment

#2280 - [CELEBORN-1182][0.4] Support application dimension ActiveConnectionCount metric to record the number of registered connections for each application

Pull Request - State: closed - Opened by SteNicholas 8 months ago - 2 comments

#2279 - [CELEBORN-1174][0.4] Introduce application dimension resource consumption metrics

Pull Request - State: closed - Opened by SteNicholas 8 months ago - 2 comments

#2278 - [CELEBORN-1260] Improve Spark Configuration of Deploy Spark client for deployment document

Pull Request - State: closed - Opened by SteNicholas 8 months ago

#2277 - [MINOR]Fix typos in profile name when checking dependencies

Pull Request - State: closed - Opened by CodingCat 8 months ago - 2 comments

#2276 - [CELEBORN-1259] Improve the default gracePeriod of ThreadUtils#shutdown

Pull Request - State: closed - Opened by SteNicholas 8 months ago - 1 comment

#2275 - [BUG] spark procedure occur Shuffle data lost for shuffle xx partitionId xx frequently

Issue - State: open - Opened by zhengshubin 8 months ago - 1 comment
Labels: bug

#2274 - [CELEBORN-XXXX] Fix NPE occurring prior to worker registration

Pull Request - State: open - Opened by cfmcgrady 8 months ago - 1 comment

#2273 - [CELEBORN-1054] Support db based dynamic config service

Pull Request - State: closed - Opened by RexXiong 8 months ago - 5 comments

#2272 - [CELEBORN-1240][FOLLOWUP] Introduce Husky Configuration to Celeborn Web

Pull Request - State: closed - Opened by tiny-dust 8 months ago - 2 comments

#2271 - [CELEBORN-1250][FOLLOWUP] Fix license issues

Pull Request - State: closed - Opened by cfmcgrady 8 months ago - 3 comments

#2270 - [DOC] Disable dynamic allocation shuffle tracking in spark 3.5

Issue - State: closed - Opened by wForget 8 months ago - 2 comments

#2269 - [CELEBORN-1246][FOLLOWUP] Introduce OpenStreamSuccessCount, FetchChunkSuccessCount and WriteDataSuccessCount metric in Grafana dashboard

Pull Request - State: closed - Opened by SteNicholas 8 months ago

#2268 - [CELEBORN-1250][FOLLOWUP] Fix license issues

Pull Request - State: closed - Opened by FMX 8 months ago - 2 comments

#2267 - [CELEBORN-1249] Add LICENSE of Celeborn Web

Pull Request - State: closed - Opened by labbomb 8 months ago - 2 comments

#2266 - [CELEBORN-1255] Fix MR UT

Pull Request - State: closed - Opened by FMX 8 months ago - 1 comment

#2265 - [CELEBORN-1012] Add a dedicated internal port in Master to talk to Workers and other Masters

Pull Request - State: closed - Opened by otterc 8 months ago - 3 comments

#2264 - [CELEBORN-1254] PartitionFilesSorter seeks to position of each block and does not warm up for non-hdfs files

Pull Request - State: closed - Opened by SteNicholas 8 months ago - 2 comments

#2263 - [CELEBORN-1247][FOLLOWUP] Remove inexistent configs

Pull Request - State: closed - Opened by FMX 8 months ago - 1 comment

#2262 - [FLINK]Support Celeborn with Hybrid mode

Pull Request - State: closed - Opened by TanYuxin-tyx 8 months ago - 1 comment

#2261 - [CELEBORN-1253] Improve exception message of fetching chunk failure for WorkerPartitionReader

Pull Request - State: closed - Opened by SteNicholas 8 months ago - 2 comments

#2260 - [CELEBORN-1252] Fix resource consumption of worker does not update when update interval is greater than heartbeat interval

Pull Request - State: closed - Opened by AngersZhuuuu 8 months ago - 3 comments

#2259 - [CELEBORN-1250] Fix license issues for release 0.4.0

Pull Request - State: closed - Opened by FMX 8 months ago - 13 comments

#2258 - [CELEBORN-1249] Add LICENSE of Celeborn Web

Pull Request - State: closed - Opened by labbomb 8 months ago - 2 comments

#2257 - [CELEBORN-1251] Connect the server and client bootstraps to RpcEnv

Pull Request - State: closed - Opened by otterc 8 months ago - 3 comments

#2256 - [CELEBORN-1246][FOLLOWUP] Introduce OpenStreamSuccessCount, FetchChunkSuccessCount and WriteDataSuccessCount metric to expose the count of opening stream, fetching chunk and writing data successfully

Pull Request - State: closed - Opened by AngersZhuuuu 8 months ago - 1 comment

#2255 - [CELEBORN-1245] Support Celeborn Master(Leader) to manage workers

Pull Request - State: closed - Opened by RexXiong 8 months ago - 4 comments

#2254 - [CELEBORN-1248] Improve flusher's robustness

Pull Request - State: closed - Opened by FMX 8 months ago - 2 comments

#2253 - [CELEBORN-1247] Output config's alternatives to doc

Pull Request - State: closed - Opened by FMX 8 months ago - 4 comments

#2252 - [CELEBORN-1246] Introduce OpenStreamSuccessCount, FetchChunkSuccessCount and WriteDataSuccessCount metric to expose the count of opening stream, fetching chunk and writing data successfully

Pull Request - State: closed - Opened by AngersZhuuuu 8 months ago - 2 comments

#2251 - [CELEBORN-1244] Delete redundant remove operations and handle timeout requests in final check

Pull Request - State: closed - Opened by jiaoqingbo 8 months ago - 3 comments

#2250 - [CELEBORN-1240] Introduce Husky Configuration to Celeborn Web

Pull Request - State: closed - Opened by tiny-dust 8 months ago - 5 comments

#2249 - [CELEBORN-1243] Bump Flink from 1.18.0 to 1.18.1

Pull Request - State: closed - Opened by cxzl25 9 months ago - 1 comment

#2248 - [CELEBORN-1242] Unify celeborn thread name format

Pull Request - State: closed - Opened by AngersZhuuuu 9 months ago - 4 comments

#2247 - [CELEBORN-1226][FOLLOWUP] Unify creation of thread using ThreadUtils

Pull Request - State: closed - Opened by AngersZhuuuu 9 months ago - 2 comments

#2246 - [CELEBORN-1241] Introduce hot load for CelebornRackResolver

Pull Request - State: closed - Opened by leixm 9 months ago - 9 comments

#2245 - [CELEBORN-1226][BRANCH-0.4.0] Unify creation of thread using ThreadUtils

Pull Request - State: closed - Opened by AngersZhuuuu 9 months ago

#2244 - [CELEBORN-1232] Add Menu to Celeborn Web

Pull Request - State: closed - Opened by tiny-dust 9 months ago - 1 comment

#2243 - [CELEBORN-1239] Introduce default quota for DefaultQuotaManager

Pull Request - State: open - Opened by leixm 9 months ago - 5 comments

#2242 - [CELEBORN-1238] deviceCheckThreadPool is only initialized when diskCheck is enabled

Pull Request - State: closed - Opened by leixm 9 months ago - 3 comments

#2241 - [CELEBORN-1225][FOLLOWUP] Worker should build replicate factory to get client for sending replicate data

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 1 comment

#2240 - [CELEBORN-1237][METRICS] Refactor metrics name.

Pull Request - State: closed - Opened by leixm 9 months ago - 6 comments

#2239 - [CELEBORN-1236][METRICS] Celeborn add metrics about thread pool

Pull Request - State: closed - Opened by AngersZhuuuu 9 months ago - 7 comments

#2237 - [CELEBORN-1235] Start test nodes in random ports to allow multiple builds run in the same ci server

Pull Request - State: closed - Opened by CodingCat 9 months ago - 2 comments

#2236 - [MINOR] Fix config type of imagePullSecrets in values.yaml for helm

Pull Request - State: closed - Opened by dev-lpq 9 months ago - 1 comment

#2235 - [CELEBORN-1233] Treat unfound PartitionLocation as failed in Controller#commitFiles

Pull Request - State: closed - Opened by waitinfuture 9 months ago - 3 comments

#2234 - [CELEBORN-1231] Support baseline implementation of Celeborn Web

Pull Request - State: closed - Opened by labbomb 9 months ago - 2 comments

#2233 - [CELEBORN-1230] Check local disk and ignore bad disk when initializing worker storage manager

Pull Request - State: open - Opened by turboFei 9 months ago - 4 comments

#2232 - [CELEBORN-1225] Worker should build replicate factory to get client for sending replicate data

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 1 comment

#2231 - [CELEBORN-1229] Support for application registration with Celeborn Master

Pull Request - State: closed - Opened by otterc 9 months ago - 6 comments

#2230 - [CELEBORN-1228] Format the timestamp when recording worker failure

Pull Request - State: closed - Opened by turboFei 9 months ago - 1 comment

#2229 - [CELEBORN-1226] Unify creation of thread using ThreadUtils

Pull Request - State: closed - Opened by AngersZhuuuu 9 months ago - 3 comments

#2228 - [CELEBORN-1224] Make TransportMessage#type transient for backward compatibility

Pull Request - State: closed - Opened by waitinfuture 9 months ago - 2 comments

#2227 - [CELEBORN-1222] Fix Celeborn worker won't record HDFS writer

Pull Request - State: closed - Opened by FMX 9 months ago - 2 comments

#2226 - [CELEBORN-1223] Align master and worker metrics of document with MasterSource and WorkerSource

Pull Request - State: closed - Opened by SteNicholas 9 months ago

#2225 - [CELEBORN-1221] Bump Flink from 1.17.0 to 1.17.2

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 2 comments

#2224 - [CELEBORN-1220][IMPROVEMENT] Make trim logic more robust

Pull Request - State: closed - Opened by AngersZhuuuu 9 months ago - 3 comments

#2223 - [CELEBORN-1219] takeBuffer() avoid checking source.metricsCollectCriticalEnabled twice

Pull Request - State: closed - Opened by AngersZhuuuu 9 months ago - 2 comments

#2222 - [CELEBORN-1133][FOLLOWUP] Refactor FileInfo

Pull Request - State: closed - Opened by cxzl25 9 months ago

#2221 - [CELEBORN-1215] Introduce PausePushDataAndReplicateTime metric to record time for a worker to stop receiving pushData from clients and other workers

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 1 comment

#2220 - [CELEBORN-1218] Optimize dataPusher to get partitionLocationMap only once

Pull Request - State: closed - Opened by lyy-pineapple 9 months ago - 3 comments

#2219 - [CELEBORN-1217] Improve exception message of loadFileGroup for ShuffleClientImpl

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 2 comments

#2218 - [CELEBORN-1216] Resolve error occurring during distribution creation with profile `-Pspark-2.4`

Pull Request - State: closed - Opened by cfmcgrady 9 months ago - 2 comments

#2217 - [CELEBORN-1214] Introduce WriteDataHardSplitCount metric to record HARD_SPLIT partitions of PushData and PushMergedData

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 1 comment

#2216 - [CELEBORN-1209][FOLLOW] Warning Shuffle Tracking

Pull Request - State: closed - Opened by zwangsheng 9 months ago - 3 comments

#2215 - [MINOR] Update log level of PushData and PushMergedData for PushDataHandler from info to debug

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 10 comments

#2214 - [MINOR] Fix typos

Pull Request - State: closed - Opened by Radeity 9 months ago - 1 comment

#2213 - [CELEBORN-1202][FOLLOWUP] Update LICENSE and NOTICE files

Pull Request - State: closed - Opened by pan3793 9 months ago - 15 comments

#2212 - [CELEBORN-1100] Introduce ChunkStreamCount, OpenStreamFailCount metrics about opening stream of FetchHandler

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 2 comments

#2211 - [CELEBORN-1213] Add pronunciation of Celeborn in README.md

Pull Request - State: closed - Opened by albin3 9 months ago - 1 comment

#2210 - [CELEBORN-1212] Support for Anonymous SASL Mechanism

Pull Request - State: closed - Opened by otterc 9 months ago - 2 comments

#2209 - [CELEBORN-1177] OpenStream should register stream via ChunkStreamManager to close stream for ReusedExchange

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 5 comments

#2208 - Revert "[CELEBORN-1150] support io encryption for spark"

Pull Request - State: closed - Opened by FMX 9 months ago - 1 comment

#2207 - [CELEBORN-1206] Add Pre-Queue for changePartitionRequests

Pull Request - State: open - Opened by lyy-pineapple 9 months ago - 3 comments

#2206 - [CELEBORN-1211] Add extension for celeborn shuffle handler

Pull Request - State: closed - Opened by FMX 9 months ago - 7 comments

#2205 - [CELEBORN-1210] Fix potential memory leak in PartitionFilesCleaner

Pull Request - State: closed - Opened by pan3793 9 months ago - 3 comments

#2204 - [CELEBORN-1209][FOLLOW] Fix warning based on Spark Enabled ShuffleTracking after Spark 3.4

Pull Request - State: closed - Opened by zwangsheng 9 months ago - 4 comments

#2203 - [CELEBORN-1209][CLIENT] Print Warning Log if User use Celeborn with enabled Spark ShuffleTracking

Pull Request - State: closed - Opened by zwangsheng 9 months ago - 1 comment

#2202 - [CELEBORN-1208][WORKER] Unify parse uniqueId to WorkerInfo

Pull Request - State: closed - Opened by zwangsheng 9 months ago - 1 comment

#2201 - [CELEBORN-1207] SBT http repository documentation

Pull Request - State: closed - Opened by cxzl25 9 months ago - 2 comments

#2200 - [CELEBORN-1206] Add Pre-Queue for changePartitionRequests

Pull Request - State: closed - Opened by lyy-pineapple 9 months ago - 2 comments

#2199 - [CELEBORN-1205] Disable Maven local caches to improve SBT building stability

Pull Request - State: closed - Opened by FMX 9 months ago - 2 comments

#2198 - [CELEBORN-1204] Update NOTICE year 2024

Pull Request - State: closed - Opened by pan3793 9 months ago - 2 comments

#2197 - [CELEBORN-1203] Add `LicenseAndNoticeMergeStrategy` to resolve inner project LICENSE/NOTICE conflict for shaded client packaging

Pull Request - State: closed - Opened by cfmcgrady 9 months ago - 3 comments

#2196 - [CELEBORN-891] Remove pipeline feature for sort based writer

Pull Request - State: closed - Opened by pan3793 9 months ago - 2 comments

#2195 - [CELEBORN-1199][FOLLOWUP] Disabled the plugin `AddMetaInfLicenseFiles` for shaded clients

Pull Request - State: closed - Opened by cfmcgrady 9 months ago - 2 comments

#2194 - [CELEBORN-1201] Optimize memory usage of cache in partition sorter

Pull Request - State: closed - Opened by FMX 9 months ago - 2 comments

#2193 - [CELEBORN-1202] LICENSE mentions third-party components under other open source licenses

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 9 comments

#2192 - [CELEBORN-1200][CORE][WIP] Scheduler Spark Reduce Task Locality with Celeborn Worker

Pull Request - State: open - Opened by zwangsheng 9 months ago - 4 comments

#2191 - [CELEBORN-1036][FOLLOWUP] When inflightBatchesPerAddress clear, totalInflightReqs should reset

Pull Request - State: closed - Opened by lyy-pineapple 9 months ago - 1 comment

#2190 - [CELEBORN-678][FOLLOWUP] MapperAttempts for a shuffle should reply MAP_ENDED when mapper has already been ended from speculative task

Pull Request - State: closed - Opened by SteNicholas 9 months ago - 1 comment

GitHub / apache/incubator-celeborn issues and pull requests