in-effficiency in flink job like : INSERT INTO hive_table select orgId, 2.0 , pdate, '02' from users limit 10000 where user is a kafka table

Question

The system should just pick 10000 message and finish . Here is what is forever running. it taking in 78 G data and forever going. don't know if this default behavior. Also. it never commit in commit sink. Above is running on flink1.12 scala 2.12 build with all hive 3.1.2

enter image description here

Answer 1

The streaming file sink only commits while checkpointing. Perhaps you need to enable and configure checkpointing.