pyspark---low frequency feature processing

Low frequency features appear less frequently,I can't learn well either,Better to give bothdefault,integrated learning.

Take a feature as an example：Use for low frequenciesdefault代替,defaultParticipate in coding and training,A new value was encountered while evaluating,也可以用default来填充,This solves the problem of poor low-frequency characterization and cold start.

tmpd = [{
    'model': 'AVA', 'city': '苏州', 'y': 0}, {
    'model': 'AVA', 'city': '苏州', 'y': 0}, 
        {
    'model': 'TNY', 'city': '青岛', 'y': 0}, {
    'model': 'AVA', 'city': '青岛', 'y': 0},
        {
    'model': 'TNY', 'city': '青岛', 'y': 0}, {
    'model': 'TNY', 'city': '青岛', 'y': 0}, 
        {
    'model': 'TNY', 'city': '青岛', 'y': 0}, {
    'model': 'AVA', 'city': '上海', 'y': 0},
        {
    'model': 'Mi', 'city': '上海', 'y': 0}, {
    'model': 'Mi', 'city': '上海', 'y': 0}, 
        {
    'model': 'Mi', 'city': '上海', 'y': 0}, {
    'model': 'fla', 'city': '北京', 'y': 0}, ]
tmpd = ss.createDataFrame(tmpd)
tmpd.show()

+----+-----+---+
|city|model|  y|
+----+-----+---+
|苏州|  AVA|  0|
|苏州|  AVA|  0|
|青岛|  TNY|  0|
|青岛|  AVA|  0|
|青岛|  TNY|  0|
|青岛|  TNY|  0|
|青岛|  TNY|  0|
|上海|  AVA|  0|
|上海|   Mi|  0|
|上海|   Mi|  0|
|上海|   Mi|  0|
|北京|  fla|  0|
+----+-----+---+

• demo示例：The number of records will be less than3 & The eigenvalues ​​that do not have a click are givendefault

def row_count2(row):
    uid, y = row[0], row[1]
    clicks = sum(y)
    lens = len(y)
    pvs = lens - clicks
    return uid, pvs, clicks, lens

''' low frequency processing '''
def low_frequency(df):
    low_enc_dict = {
    'city': 3, 'model': 3}
    for c in low_enc_dict.keys():
        dfpg = df.groupby(c).agg(fn.collect_list('y').alias('y')).rdd.map(row_count2).toDF(schema=[c, 'pvs', 'clicks', 'lens'])

# print(dfpg.count())
# dfpg.orderBy(['lens', 'clicks'], ascending=[0, 0]).show(10)
# dfpg.orderBy(['lens', 'clicks'], ascending=[1, 1]).show(50)

# dfpg.filter(dfpg['lens'] <= low_enc_dict[c]).filter(dfpg['clicks'] == 0).select('lens').agg({'lens': 'sum'}).show()

        lowlst = []
        tlst = dfpg.filter(dfpg['lens'] <= low_enc_dict[c]).filter(dfpg['clicks'] == 0).select(c).collect()
        for row in tlst:
            lowlst.append(row[0])

        df = df.withColumn(c, fn.udf(lambda x: 'default' if x in lowlst else x)(fn.col(c)))

# print(c, len(lowlst), df.filter(df[c] == 'default').count())
        
    return df
    

tmpd = low_frequency(df=tmpd)
tmpd.show()

+-------+-------+---+
|   city|  model|  y|
+-------+-------+---+
|default|    AVA|  0|
|default|    AVA|  0|
|   青岛|    TNY|  0|
|   青岛|    AVA|  0|
|   青岛|    TNY|  0|
|   青岛|    TNY|  0|
|   青岛|    TNY|  0|
|   上海|    AVA|  0|
|   上海|default|  0|
|   上海|default|  0|
|   上海|default|  0|
|default|default|  0|
+-------+-------+---+