Performance is not improved for too complex query

[njalan]

In my testing most of the jobs are improved compared without celeborn But there is no improve on my most slow and complex query.

Any other information required for it?
How to troubleshoot it ?

[FMX]

Hi @njalan . Can you share your cluster information and your workload information?
You'll have to find out the most time-consuming part of your query. If the shuffle-related process is the one, Celeborn will help a lot.

[njalan]

@FMX Spark version is 3.0.1 and hadoop is 3.2.2 with 17 work nodes.
What is any other cluster information you need?
How to get workload information? what is the metrics for it?

[njalan]

@FMX How to find ind out the most time-consuming part of query? Do you have any links for it?