Recommendation systems: feature engineering, common features

一、特征工程的重要性

• Feature engineering can make machine learning models achieve better results

Features commonly used in recommender systems

• 用户行为信息
• 属性、标签信息（不容易获取）
• 用户关系信息
• 内容信息
• 上下文信息

二、Inadequate original features

• does not belong to the unified dimension
• 信息冗余
• There are non-quantitative qualitative features
• 存在缺失值

三、A common approach to feature engineering

• 标准化
  • It is more suitable for data that is normally distributed（如价格）
  • 对异常值不敏感
• 归一化
  • Suitable for data whose distribution is uncertain（Such as dummy encoding backend classification data）
  • 对异常值较为敏感
• 二值化
  • Transform qualitative features into quantitative features
• 哑编码
  • Convert discrete attribute classification features to 0、1向量
• 缺失值补全
  • Commonly used supplement0、平均值、median and other methods

四、特征选择

Apache Spark

• An open source distributed computing framework
  • 计算速度快：相对于Hadoop有最多100倍的提升 - Powerful cache design：Provides memory through a simple interface+硬盘缓存
  • 部署灵活：支持YARN,k8s等集群管理工具 - 实时性高：Provides tools specific to stream computing
  • 通用性高：提供多种语言APIand various business abstractions
• RDD
  • Resilient Distributed Dataset
  • Resilient: Good fault tolerance and automatic error recovery
  • Distributed：天生的分布式
  • Dataset：Provide uniformity to users、Distributed transparent programming interface

行为数据采集

• Data generated when a user interacts with a product,如点赞、收藏、浏览
• Usually uploaded by the client
• 为何使用KafkaProcess behavioral data？ - 解耦：Message producers and consumers can work independently of each other
  • 拓展性：To cope with the rapid expansion of the number of users, it can efficiently expand the capacity
  • 削峰填谷：Effectively ensure the smooth distribution of traffic during the event
  • 异步通信：Suitable for processing behavioral data
• Kafka核心概念
  • Broker：集群中的服务器
  • Topic：The logical category of the message
  • Partition：topicphysical storage unit - Producer\Consumer：消息生产、消费者 - Consumer Group：消费者群组