“英特尔创新大师杯”深度学习挑战赛 赛道3：CCKS2021中文NLP地址相关性任务

Related tags

Deep Learningccks2021-track3-top1
Overview

ccks2021-track3

CCKS2021中文NLP地址相关性任务-赛道三-冠军方案

团队：我的加菲鱼- wodejiafeiyu

初赛第二/复赛第一/决赛第一

前言

19年开始，陆陆续续参加了一些比赛，拿到过一些top，比较懒一直都没分享过，这次比较幸运又拿了top1，打算分享下

分类的任务用这个框架基本都能top10，之前个人参加的《全球人工智能大赛-赛道一》和这个也差不多，只是有些具体任务的trick不同

比赛主页

https://tianchi.aliyun.com/competition/entrance/531901/introduction

环境

  • torch==1.6.0
  • transformers=3.0.2

预训练模型

- nezha-base 
- nezha-wwm
- macbert
  • 下载预训练模型，放到文件夹user_data/model_param/pretrain_model_param/下，文件夹和模型名字一一对应

全流程脚本

sh run.sh

核心思路

  • 预训练mlm中的mask策略使用ngram-mask，相比原始的动态mask，提升了预训练难度
  • 标签也包含语义信息，预训练部分融入标签信息，提升预训练效果
  • 混合精度预训练，损失一部分精度，提升整体的训练速度，实际测试结果精度损失不大，速度提升明显
  • 对抗训练，已经是一个比较常用的trick了
  • 后12层加上embedding的cls动态加权平均
  • multi-sample dropout
  • 推理的时候，阈值搜索
  • 三折交叉验证，每折使用不同的随机数种子使用dynamic pad
  • 加权平均，模型融合

总结

模块 提分点分析 提升
ngram-mask 相比于单个字的遮蔽，ngram-mask加大了预训练任务的难度，从而提升效果 2.6个千分点
融入标签信息预训练 标签也含有语义信息，模型学习的更多 1个千分点
后12层加上embedding的cls动态加权平均+multi-sample dropout 加权平均能增强向量的语义表征能力从而提升效果。multi-sample dropout能加速训练，增强泛化能力 3.3个千分点
对抗训练fgm 生成对抗样本，对抗样本的训练，增加模型泛化能力 2.2个千分点
多分类阈值搜索 缩放系数使得f1最优 0.7个千分点
模型融合 提升模型的鲁棒性和泛化能力 1.5个千分点
动态pad和预训练混合精度训练和每折使用不同的随机数种子 提升模型训练速度，不同的随机数种子加大了模型的差异，提升融合的效果
Owner
shaochenjie
shaochenjie
GitHub Repository
Fast RFC3339 compliant Python date-time library

udatetime: Fast RFC3339 compliant date-time library Handling date-times is a painful act because of the sheer endless amount of formats used by people

Simon Pirschel 235 Oct 25, 2022
METER: Multimodal End-to-end TransformER

METER Code and pre-trained models will be publicized soon. Citation @article{dou2021meter, title={An Empirical Study of Training End-to-End Vision-a

Zi-Yi Dou 257 Jan 06, 2023
A CNN model to detect hand gestures.

Software Used python - programming language used, tested on v3.8 miniconda - for managing virtual environment Libraries Used opencv - pip install open

Shivanshu 6 Jul 14, 2022
Improving Object Detection by Label Assignment Distillation

Improving Object Detection by Label Assignment Distillation This is the official implementation of the WACV 2022 paper Improving Object Detection by L

Cybercore Co. Ltd 51 Dec 08, 2022
Official repository of the paper "GPR1200: A Benchmark for General-PurposeContent-Based Image Retrieval"

GPR1200 Dataset GPR1200: A Benchmark for General-Purpose Content-Based Image Retrieval (ArXiv) Konstantin Schall, Kai Uwe Barthel, Nico Hezel, Klaus J

Visual Computing Group 16 Nov 21, 2022
RL algorithm PPO and IRL algorithm AIRL written with Tensorflow.

RL algorithm PPO and IRL algorithm AIRL written with Tensorflow. They have a parallel sampling feature in order to increase computation speed (especially in high-performance computing (HPC)).

Fangjian Li 3 Dec 28, 2021
System Combination for Grammatical Error Correction Based on Integer Programming

System Combination for Grammatical Error Correction Based on Integer Programming This repository contains the code and scripts that implement the syst

NUS NLP Group 0 Mar 29, 2022
Detecting drunk people through thermal images using Deep Learning (CNN)

Drunk Detection CNN Detecting drunk people through thermal images using Deep Learning (CNN) Dataset We used thermal images provided by Electronics Lab

Giacomo Ferretti 3 Oct 27, 2022
An ML & Correlation platform for transforming disparate data points of interest into usable intelligence.

SSIDprobeCollector An ML & Correlation platform for transforming disparate data points of interest into usable intelligence. At a High level the platf

Bill Reyor 1 Jan 30, 2022
Minimal PyTorch implementation of YOLOv3

A minimal PyTorch implementation of YOLOv3, with support for training, inference and evaluation.

Erik Linder-Norén 6.9k Dec 29, 2022
The Pytorch implementation for "Video-Text Pre-training with Learned Regions"

Region_Learner The Pytorch implementation for "Video-Text Pre-training with Learned Regions" (arxiv) We are still cleaning up the code further and pre

Rui Yan 0 Mar 20, 2022
Fully convolutional networks for semantic segmentation

FCN-semantic-segmentation Simple end-to-end semantic segmentation using fully convolutional networks [1]. Takes a pretrained 34-layer ResNet [2], remo

Kai Arulkumaran 186 Dec 25, 2022
Rendering color and depth images for ShapeNet models.

Color & Depth Renderer for ShapeNet This library includes the tools for rendering multi-view color and depth images of ShapeNet models. Physically bas

Yinyu Nie 41 Dec 19, 2022
Creating Artificial Life with Reinforcement Learning

Although Evolutionary Algorithms have shown to result in interesting behavior, they focus on learning across generations whereas behavior could also be learned during ones lifetime.

Maarten Grootendorst 49 Dec 21, 2022
DISTIL: Deep dIverSified inTeractIve Learning.

DISTIL: Deep dIverSified inTeractIve Learning. An active/inter-active learning library built on py-torch for reducing labeling costs.

decile-team 110 Dec 06, 2022
Implementation of accepted AAAI 2021 paper: Deep Unsupervised Image Hashing by Maximizing Bit Entropy

Deep Unsupervised Image Hashing by Maximizing Bit Entropy This is the PyTorch implementation of accepted AAAI 2021 paper: Deep Unsupervised Image Hash

62 Dec 30, 2022
Re-implementation of the vector capsule with dynamic routing

VectorCapsule Re-implementation of the vector capsule with dynamic routing We implement the vector capsule and dynamic routing via graph neural networ

ZhenchaoTang 10 Feb 10, 2022
Additional code for Stable-baselines3 to load and upload models from the Hub.

Hugging Face x Stable-baselines3 A library to load and upload Stable-baselines3 models from the Hub. Installation With pip Examples [Todo: add colab t

Hugging Face 34 Dec 10, 2022
Official implementation for CVPR 2021 paper: Adaptive Class Suppression Loss for Long-Tail Object Detection

Adaptive Class Suppression Loss for Long-Tail Object Detection This repo is the official implementation for CVPR 2021 paper: Adaptive Class Suppressio

CASIA-IVA-Lab 67 Dec 04, 2022
Fast and scalable uncertainty quantification for neural molecular property prediction, accelerated optimization, and guided virtual screening.

Evidential Deep Learning for Guided Molecular Property Prediction and Discovery Ava Soleimany*, Alexander Amini*, Samuel Goldman*, Daniela Rus, Sangee

Alexander Amini 75 Dec 15, 2022