apache/celeborn issues and pull requests

#2860 - [FEATURE] Support web ui

Issue - State: open - Opened by melin 16 days ago

#2859 - [CELEBORN-1490][CIP-6] Add Flink Hybrid Shuffle IT test cases

Pull Request - State: open - Opened by reswqa 16 days ago

#2858 - [CELEBORN-1677][BUILD] Update SCM information for SBT build configuration

Pull Request - State: closed - Opened by cfmcgrady 16 days ago - 1 comment

#2857 - [CELEBORN-1675] User with zero quota should not be allowed to submit any shuffle to cluster.

Pull Request - State: open - Opened by Z1Wu 17 days ago - 3 comments

#2856 - [CELEBORN-1636][FOLLOWUP] Cache available workers in Master

Pull Request - State: open - Opened by zaynt4606 17 days ago

#2855 - [CELEBORN-1673] Support retry create client

Pull Request - State: open - Opened by RexXiong 17 days ago

#2854 - [CELEBORN-1671] CelebornShuffleReader will try replica if create client failed

Pull Request - State: open - Opened by FMX 17 days ago

#2853 - [CELEBORN-1674] Fix reader thread name of MapPartitionData

Pull Request - State: open - Opened by SteNicholas 17 days ago

#2852 - [CELEBORN-1636][FOLLOWUP] Dynamic resources will only be utilized in case of candidates shortages

Pull Request - State: open - Opened by zaynt4606 17 days ago

#2851 - [CELEBORN-1672] Bump Spark from 3.4.3 to 3.4.4

Pull Request - State: open - Opened by SteNicholas 17 days ago

#2850 - [CELEBORN-1642][CIP-11] Support multiple worker tags

Pull Request - State: closed - Opened by s0nskar 21 days ago - 1 comment

#2849 - [CELEBORN-1670] Avoid swallowing InterruptedException in ShuffleClientImpl

Pull Request - State: open - Opened by jiang13021 21 days ago - 1 comment

#2848 - [CELEBORN-1516][FOLLOWUP] Support reset method for DynamicConfigServiceFactory

Pull Request - State: open - Opened by s0nskar 21 days ago - 2 comments

#2847 - [CELEBORN-1669] Fix NullPointerException for PartitionFilesSorter#updateSortedShuffleFiles after cleaning up expired shuffle key

Pull Request - State: closed - Opened by SteNicholas 21 days ago

#2846 - [CELEBORN-1668] Fix NPE when handle closed file writers

Pull Request - State: closed - Opened by FMX 22 days ago

#2845 - [CELEBORN-1664][0.5] Fix secret fetch failures after LEADER master failover

Pull Request - State: closed - Opened by YutingWang98 22 days ago - 2 comments

#2844 - [CELEBORN-1619][CIP-11] Integrate tags manager with config service

Pull Request - State: open - Opened by s0nskar 22 days ago

#2843 - [CELEBORN-1667] Fix NPE & LEAK occurring prior to worker registration

Pull Request - State: closed - Opened by zhaostu4 22 days ago - 3 comments

#2842 - [CELEBORN-1666] Bump scala-protoc from 1.0.6 to 1.0.7

Pull Request - State: closed - Opened by cfmcgrady 22 days ago - 1 comment

#2841 - [MINOR] Fix typo in ExceptionUtils

Pull Request - State: closed - Opened by jiang13021 22 days ago

#2840 - [CELEBORN-1665][0.5] CommitHandler should process CommitFilesResponse with COMMIT_FILE_EXCEPTION status

Pull Request - State: closed - Opened by SteNicholas 23 days ago

#2839 - [CELEBORN-1660] Using map for workers to find worker fast and WokerInfo pool to reuse objects

Pull Request - State: open - Opened by turboFei 23 days ago - 4 comments

#2838 - [CELEBORN-1665] CommitHandler should process CommitFilesResponse with COMMIT_FILE_EXCEPTION status

Pull Request - State: closed - Opened by SteNicholas 23 days ago - 1 comment

#2837 - [CELEBORN-1663][FOLLOWUP] Only register appShuffleDeterminate if stage using celeborn for shuffle

Pull Request - State: closed - Opened by s0nskar 23 days ago

#2836 - [CELEBORN-1444][FOLLOWUP] Add IsDecommissioningWorker to celeborn dashboard

Pull Request - State: open - Opened by s0nskar 24 days ago - 2 comments

#2835 - [CELEBORN-1636] Client supports dynamic update of Worker resources on the server

Pull Request - State: closed - Opened by zaynt4606 24 days ago - 1 comment

#2834 - [CELEBORN-1649]: Bumping up maven to 3.9.9

Pull Request - State: closed - Opened by avishnus 24 days ago - 1 comment

#2833 - [CELEBORN-1662] Handle PUSH_DATA_FAIL_PARTITION_NOT_FOUND in getPushDataFailCause

Pull Request - State: closed - Opened by jiang13021 24 days ago - 1 comment

#2832 - [CELEBORN-1663] Only register appShuffleDeterminate if stage using celeborn for shuffle

Pull Request - State: closed - Opened by s0nskar 24 days ago - 2 comments

#2831 - [CELEBORN-1661] Make sure that the sortedFilesDb is initialized successfully when worker enable graceful shutdown

Pull Request - State: closed - Opened by cxzl25 24 days ago - 1 comment

#2830 - [CELEBORN-1530] support MPU for S3

Pull Request - State: open - Opened by zhaohehuhu 24 days ago - 2 comments

#2829 - [CELEBORN-1632] Support to apply ratis local raft_meta_conf command with RESTful api

Pull Request - State: closed - Opened by turboFei 26 days ago - 4 comments

#2828 - [CELEBORN-1629][FOLLOWUP] Fix broken RESTful api link

Pull Request - State: closed - Opened by turboFei 26 days ago

#2827 - [CELEBORN-1660] Only count the available workers device free capacity

Pull Request - State: open - Opened by turboFei 27 days ago - 2 comments

#2826 - [CELEBORN-1664] Fix secret fetch failures after LEADER master failover

Pull Request - State: closed - Opened by YutingWang98 27 days ago - 9 comments

#2825 - [CELEBORN-1627][FOLLOWUP] Fix typo for metrics_SlotsAllocated_increas_1h

Pull Request - State: closed - Opened by turboFei 27 days ago

#2823 - [CELEBORN-1659][FOLLOWUP] Dockerfile should support copying CLI jars

Pull Request - State: closed - Opened by SteNicholas 27 days ago - 1 comment

#2822 - [CELEBORN-1659] Fix sbt make-distribution for cli

Pull Request - State: closed - Opened by akpatnam25 27 days ago - 1 comment

#2821 - [CELEBORN-1658] Add Git Commit Info and Build JDK Spec to sbt Manifest

Pull Request - State: closed - Opened by cfmcgrady 28 days ago - 2 comments

#2820 - [CELEBORN-1490][CIP-6] Impl worker read process in Flink Hybrid Shuffle

Pull Request - State: closed - Opened by reswqa 28 days ago - 2 comments

#2819 - [WIP][CELEBORN-1577][Phase2] QuotaManager should support interrupt shuffle.

Pull Request - State: open - Opened by leixm 28 days ago

#2818 - [CELEBORN-1657] Throw exception to client when exception occurred in reading sort shuffle files

Pull Request - State: closed - Opened by FMX 28 days ago

#2817 - [CELEBORN-1487][Phase2] CongestionController support dynamic config

Pull Request - State: closed - Opened by leixm 29 days ago - 1 comment

#2816 - [CELEBORN-1656] Remove duplicate UserProduceSpeed metrics

Pull Request - State: closed - Opened by leixm 29 days ago

#2815 - [CELEBORN-1655] Fix read buffer dispatcher thread terminate unexpectedly

Pull Request - State: closed - Opened by FMX 29 days ago - 2 comments

#2814 - [CELEBORN-1654] Fix flaky test TestCongestionController#testUserMetrics

Pull Request - State: closed - Opened by leixm 29 days ago - 1 comment

#2813 - [CELEBORN-1413] Support Spark 4.0

Pull Request - State: open - Opened by FMX about 1 month ago

#2812 - [CELEBORN-1651] Support ratio threshold of unhealthy disks for excluding worker

Pull Request - State: closed - Opened by SteNicholas about 1 month ago - 1 comment

#2811 - [CELEBORN-1652] Throw TransportableError for failure of sending PbReadAddCredit to avoid flink task get stuck

Pull Request - State: closed - Opened by codenohup about 1 month ago - 3 comments

#2810 - [CELEBORN-1648] Refine AppUniqueId with UUID suffix

Pull Request - State: closed - Opened by chenkovsky about 1 month ago - 1 comment

#2809 - [CELEBORN-1646] Catch exception of Files#getFileStore for DeviceMonitor and StorageManager for input/ouput error

Pull Request - State: closed - Opened by SteNicholas about 1 month ago

#2808 - All ratis apis

Pull Request - State: closed - Opened by turboFei about 1 month ago

#2807 - [CELEBORN-1645] Introduce ShuffleFallbackPolicy to support custom implementation of shuffle fallback policy for CelebornShuffleFallbackPolicyRunner

Pull Request - State: closed - Opened by SteNicholas about 1 month ago - 2 comments

#2806 - [CELEBORN-1644] Optimize handle merged data on stage end

Pull Request - State: closed - Opened by FMX about 1 month ago

#2805 - [CELEBORN-1643] DataPusher handle InterruptedException

Pull Request - State: closed - Opened by cxzl25 about 1 month ago - 2 comments

#2804 - [CELEBORN-1630] Support to apply ratis peer operation with RESTful api

Pull Request - State: closed - Opened by turboFei about 1 month ago

#2803 - [CELEBORN-1631] Support to apply ratis snapshot operation with RESTful api

Pull Request - State: closed - Opened by turboFei about 1 month ago - 1 comment

#2802 - [CELEBORN-1640] NettyMemoryMetrics supports numHeapArenas, numDirectArenas, tinyCacheSize, smallCacheSize, normalCacheSize, numThreadLocalCaches and chunkSize

Pull Request - State: closed - Opened by SteNicholas about 1 month ago - 1 comment

#2801 - [CELEBORN-1577][Phase1] Storage quota should support interrupt shuffle.

Pull Request - State: open - Opened by leixm about 1 month ago - 3 comments

#2800 - [CELEBORN-1639] Fix SlotsAllocatorRackAwareSuiteJ UT

Pull Request - State: closed - Opened by wankunde about 1 month ago

#2799 - [CELEBORN-1638] Improve the slots allocator performance

Pull Request - State: closed - Opened by wankunde about 1 month ago - 1 comment

#2798 - [CELEBORN-1637] Enhance config to bypass memory check for partition file sorter

Pull Request - State: closed - Opened by FMX about 1 month ago - 6 comments

#2797 - [CELEBORN-1487][Phase1] CongestionController support control traffic by user/worker traffic speed

Pull Request - State: closed - Opened by leixm about 1 month ago - 2 comments

#2796 - [CELEBORN-474][FOLLOWUP] Use JavaUtils#newConcurrentHashMap to speed up ConcurrentHashMap#computeIfAbsent

Pull Request - State: closed - Opened by SteNicholas about 1 month ago - 1 comment

#2795 - [CELEBORN-1636] The client supports dynamic updates of worker resources on the server

Pull Request - State: closed - Opened by zaynt4606 about 1 month ago

#2794 - [CELEBORN-1615][FOLLOWUP][0.5] Sleep for http server initialization in UT

Pull Request - State: closed - Opened by turboFei about 1 month ago - 1 comment

#2793 - [CELEBORN-1615][FOLLOWUP] Start connector before starting server

Pull Request - State: closed - Opened by turboFei about 1 month ago

#2792 - Check 0.5 ut

Pull Request - State: closed - Opened by turboFei about 1 month ago

#2791 - [CELEBORN-1615][FOLLOWUP][0.5] Sleep for http server initialization

Pull Request - State: closed - Opened by turboFei about 1 month ago

#2790 - [CELEBORN-1593][FOLLOWUP] Support master --show-manual-excluded-workers command to show manual excluded workers

Pull Request - State: closed - Opened by SteNicholas about 1 month ago - 1 comment

#2789 - [CELEBORN-1569][FOLLOWUP] Identify the unit test log name with java/scala version

Pull Request - State: closed - Opened by turboFei about 1 month ago

#2788 - [CELEBORN-1540] Simplify some code in Tez and improve integration testing

Pull Request - State: closed - Opened by GH-Gloway about 1 month ago - 2 comments

#2787 - [CELEBORN-1635] Introduce Blaze support document

Pull Request - State: closed - Opened by SteNicholas about 1 month ago

#2786 - [CELEBORN-1490][CIP-6] Introduce tier consumer for hybrid shuffle

Pull Request - State: closed - Opened by reswqa about 1 month ago - 2 comments

#2785 - [CELEBORN-1624] Add tez shade module

Pull Request - State: closed - Opened by Jay-ju about 1 month ago - 1 comment

#2784 - [WIP][CELEBORN-1634] implement queue time/processing time metrics for rpc framework

Pull Request - State: open - Opened by ErikFang about 1 month ago

#2780 - [CELEBORN-1633] Return more raft group information

Pull Request - State: closed - Opened by turboFei about 1 month ago

#2779 - [CELEBORN-1629] Support to apply ratis election operation with RESTful api

Pull Request - State: closed - Opened by turboFei about 1 month ago - 5 comments

#2777 - [CELEBORN-1627] Introduce `instance` variable for celeborn dashboard to filter metrics

Pull Request - State: closed - Opened by turboFei about 1 month ago - 2 comments

#2776 - [CELEBORN-1451] HPA support

Pull Request - State: open - Opened by lianneli about 2 months ago - 5 comments
Labels: stale

#2771 - [CELEBORN-1626] group mapTask

Pull Request - State: open - Opened by zaynt4606 about 2 months ago

#2770 - [CELEBORN-1620][CIP-11] Support passing worker tags via RequestSlots message

Pull Request - State: closed - Opened by s0nskar about 2 months ago - 3 comments

#2767 - [CELEBORN-1624] Refine tez client shade package

Pull Request - State: closed - Opened by Jay-ju about 2 months ago - 2 comments

#2766 - [CELEBORN-1617][CIP-11] Support workers tags in FS Config Service

Pull Request - State: closed - Opened by s0nskar about 2 months ago - 2 comments

#2764 - [CELEBORN-1615] Start the http server after all handlers added

Pull Request - State: closed - Opened by turboFei about 2 months ago - 2 comments

#2763 - [CELEBORN-1614] Remove unused dependencies from openapi-client, spi and cli modules

Pull Request - State: closed - Opened by turboFei about 2 months ago - 2 comments
Labels: stale

#2758 - [CELEBORN-1599] Container Info REST API

Pull Request - State: closed - Opened by akpatnam25 about 2 months ago - 10 comments

#2757 - [CELEBORN-1540] Add patch to customize tez main class.

Pull Request - State: closed - Opened by FMX about 2 months ago

#2746 - [CELEBORN-1601] Support revise lost shuffles

Pull Request - State: closed - Opened by FMX about 2 months ago - 6 comments

#2727 - [CELEBORN-1595] Support quota low watermark for checking quota available

Pull Request - State: open - Opened by s0nskar 2 months ago - 5 comments

#2723 - [CELEBORN-1586] Add available workers Metrics

Pull Request - State: closed - Opened by zaynt4606 2 months ago - 1 comment

#2703 - [CELEBORN-1521][FOLLOWUP] Using AuthenticationException instead of SecurityException

Pull Request - State: closed - Opened by turboFei 3 months ago - 4 comments

#2701 - [CELEBORN-1574] Speed up unregister shuffle by batch processing

Pull Request - State: closed - Opened by zaynt4606 3 months ago - 2 comments

#2697 - [CELEBORN-1571] Fix flaky test - pushdata timeout will add to pushExcludedWorker

Pull Request - State: closed - Opened by cxzl25 3 months ago - 2 comments

#2672 - [CELEBORN-1536] Add option to toggle between human friendly vs single line logging

Pull Request - State: closed - Opened by s0nskar 3 months ago - 8 comments
Labels: stale

#2654 - [CELEBORN-1532][HELM] Make helm charts more customizable

Pull Request - State: closed - Opened by ChenYi015 4 months ago - 5 comments
Labels: stale

#2626 - [CELEBORN-1510] Partial task unable to switch to the replica

Pull Request - State: closed - Opened by zhaostu4 4 months ago - 6 comments
Labels: stale

#2613 - [CELEBORN-1497] Optimize read buffer dispatcher to retain buffers until worker has memory pressure

Pull Request - State: closed - Opened by FMX 4 months ago - 4 comments
Labels: stale

#2373 - [WIP][CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files

Pull Request - State: open - Opened by wangshengjie123 8 months ago - 42 comments

#2140 - Who is using Apache Celeborn?

Issue - State: open - Opened by SteNicholas 11 months ago - 25 comments

GitHub / apache/celeborn issues and pull requests