Record an oom exception in production

background

In order to make business acceptance sql, I spent a week doing sql test , There is no problem running in the test environment , But once on production , Previously successful code , The whole army was basically destroyed . The data warehouse structure of the company is Hive+Doris In the form of , It means that the warehouse is Hive On the establishment of , final ads Layer to map to Doris in , Sail soft by reading Doris To visually display the data in .

problem

In the test environment Hive The acceptance of statement creation is basically completed , But when it comes to the production environment ,hive The table in has no content ,dws And dwt Data ok , only ads Layer of hour There is no data here . Because the dispatcher did not report an error , So I passed through the server hive -f XX.sql Go for a run , If you do, you will report an error , Basically, polling exceptions, etc .

solve

At every sql Add five parameters to the file , Set up hive Of memory .

set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.max.dynamic.partitions.pernode=10000000;
set hive.exec.max.dynamic.partitions =10000000;
set hive.exec.max.created.files =10000000;
set hive.auto.convert.join=false;

Experience

The production environment has more data than the test environment , Easy to photograph OOM. Or the hive To maximize the setting of parameters .