How are the open source Netease cloud music API projects implemented?

Last article You deserve this high-value open-source third-party Netease cloud music player This paper introduces an open source third-party Netease cloud music player , In this article, let's have a detailed understanding of Netease cloud music api project NeteaseCloudMusicApi Implementation principle of .

NeteaseCloudMusicApi Use Node.js Development , There are two main frameworks and libraries , One Web Application development framework Express, A request Library Axios, You should be familiar with these two, but I don't want to introduce them more .

establish express application

The entry file of the project is /app.js：

async function start() {
    
  require('./server').serveNcmApi({
    
    checkVersion: true,
  })
}
start()

Called /server.js Of documents serveNcmApi Method , Let's go to this file ,serveNcmApi The simplified method is as follows ：

async function serveNcmApi(options) {
    
    const port = Number(options.port || process.env.PORT || '3000')
    const host = options.host || process.env.HOST || ''
    const app = await consturctServer(options.moduleDefs)
    const appExt = app
    appExt.server = app.listen(port, host, () => {
    
        console.log(`server running @ http://${
      host ? host : 'localhost'}:${
      port}`)
    })

    return appExt
}

It mainly starts listening to the specified port , So the main logic of creating applications is consturctServer Method ：

async function consturctServer(moduleDefs) {
    
    //  Create an application 
    const app = express()

    //  Set to true, The client's IP The address is understood as X-Forwarded-* The leftmost entry in the header 
    app.set('trust proxy', true)

    /** *  To configure CORS &  Pre inspection request  */
    app.use((req, res, next) => {
    
        if (req.path !== '/' && !req.path.includes('.')) {
    
            res.set({
    
                'Access-Control-Allow-Credentials': true, //  Cross domain , Allow the client to carry authentication information , such as cookie, meanwhile , The front end also needs to be set when sending requests withCredentials: true
                'Access-Control-Allow-Origin': req.headers.origin || '*', //  Domain names that allow cross domain requests , Set to * On behalf of allowing all domain names 
                'Access-Control-Allow-Headers': 'X-Requested-With,Content-Type', //  Used for pre inspection request (options) List the allowed custom headers on the server , If the request sent by the front end contains a custom request header , And the header is not included in Access-Control-Allow-Headers in , Then the request cannot be successfully initiated 
                'Access-Control-Allow-Methods': 'PUT,POST,GET,DELETE,OPTIONS', //  It is ideal to set the request method allowed by cross domain requests 
                'Content-Type': 'application/json; charset=utf-8', //  Set the type and code of response data 
            })
        }
        // OPTIONS Request for pre inspection , Complex requests will send a pre check request before sending the real request , Get server support for Access-Control-Allow-xxx Related information , Determine whether it is necessary to send a real request later , Return status code 204 On behalf of the successful request , But there's no content 
        req.method === 'OPTIONS' ? res.status(204).end() : next()
    })
    // ...
}

First, I created a Express application , Then set it as a trust agent , stay Express get ip Usually by req.ip and req.ips,trust proxy The default value is false, In this case req.ips Value is empty , When set to true when ,req.ip The value of will start from the request header X-Forwarded-For Take the leftmost value ,req.ips Will contain X-Forwarded-For All of the head ip Address .

X-Forwarded-For The format of the header is as follows ：

X-Forwarded-For: client1, proxy1, proxy2

Value through a comma + Space The multiple ip Address discrimination , The leftmost client1 Is the original client ip Address , Every time the proxy server successfully receives a request , Just put Source of the request ip Address Add to the right .

Take the example above , This request passed through two proxy servers ：proxy1 and proxy2. Request by the client1 issue , here XFF It's empty. , here we are proxy1 when ,proxy1 hold client1 Add to XFF in , Then the request is sent to proxy2, adopt proxy2 When ,proxy1 Be added to XFF in , Then the request is sent to the final server , After arrival proxy2 Be added to XFF in .

But it's very easy to forge this field , So when contemporary reason is not believable , This field is not necessarily reliable , But normally XFF Last of ip The address must be the last proxy server ip Address , This will be more reliable .

Then set the cross domain response header , The setting here is the key to allowing websites with different domain names to request successfully .

continue ：

async function consturctServer(moduleDefs) {
    
    // ...
    /** *  analysis Cookie */
    app.use((req, _, next) => {
    
        req.cookies = {
    }
        //;(req.headers.cookie || '').split(/\s*;\s*/).forEach((pair) => { // Polynomial regular expression //
        //  Read from the request header cookie,cookie The format is ：name=value;name2=value2..., So first according to ; Cut into arrays 
        ;(req.headers.cookie || '').split(/;\s+|(?<!\s)\s+$/g).forEach((pair) => {
    
            let crack = pair.indexOf('=')
            //  Skip directly without value 
            if (crack < 1 || crack == pair.length - 1) return
            //  take cookie Save to cookies On the object 
            req.cookies[decode(pair.slice(0, crack)).trim()] = decode(
                pair.slice(crack + 1),
            ).trim()
        })
        next()
    })

    /** *  Request body parsing and file upload processing  */
    app.use(express.json())
    app.use(express.urlencoded({
     extended: false }))
    app.use(fileUpload())

    /** *  take public The files in the directory are provided as static files  */
    app.use(express.static(path.join(__dirname, 'public')))

    /** *  Cache request , Within two minutes, the same request will read data from the cache , Will not send requests to Netease cloud music server  */
    app.use(cache('2 minutes', (_, res) => res.statusCode === 200))
    // ...
}

Next, I registered some middleware , Used to resolve cookie、 Processing request body, etc , In addition, interface caching is also done , Prevent too frequent requests for Netease cloud music servers from being blocked .

continue ：

async function consturctServer(moduleDefs) {
    
    // ...
    /** *  Special routing  */
    const special = {
    
        'daily_signin.js': '/daily_signin',
        'fm_trash.js': '/fm_trash',
        'personal_fm.js': '/personal_fm',
    }

    /** *  load /module All modules in the directory , Each module corresponds to an interface  */
    const moduleDefinitions =
          moduleDefs ||
          (await getModulesDefinitions(path.join(__dirname, 'module'), special))
    // ...
}

Next, load /module All modules in the directory ：

Each module represents a request for Netease cloud music interface , For example, get album details album_detail.js：

Module loading method getModulesDefinitions as follows ：

async function getModulesDefinitions(
  modulesPath, specificRoute, doRequire = true,
) {
    
  const files = await fs.promises.readdir(modulesPath)
  const parseRoute = (fileName) =>
    specificRoute && fileName in specificRoute
      ? specificRoute[fileName]
      : `/${
      fileName.replace(/\.js$/i, '').replace(/_/g, '/')}`
  //  Traverse all the files in the directory 
  const modules = files
    .reverse()
    .filter((file) => file.endsWith('.js'))//  Filter out js file 
    .map((file) => {
    
      const identifier = file.split('.').shift()//  Module identifier 
      const route = parseRoute(file)//  The route corresponding to the module 
      const modulePath = path.join(modulesPath, file)//  The module path 
      const module = doRequire ? require(modulePath) : modulePath//  Load module 

      return {
     identifier, route, module }
    })

  return modules
}

As just now album_detail.js Module as an example , The data returned is as follows ：

{
     
    identifier: 'album_detail', 
    route: '/album/detail', 
    module: () => {
    /* Module content */}
}

The next step is to register routes ：

async function consturctServer(moduleDefs) {
     
    // ...
    for (const moduleDef of moduleDefinitions) {
    
        //  Registered routing 
        app.use(moduleDef.route, async (req, res) => {
    
            // cookie You can also query parameters 、 The request body is uploaded 
            ;[req.query, req.body].forEach((item) => {
    
                if (typeof item.cookie === 'string') {
    
                    //  take cookie The string is converted to json type 
                    item.cookie = cookieToJson(decode(item.cookie))
                }
            })

            //  hold cookie、 Query parameters 、 Request header 、 Documents are all integrated , Pass it to each module as a parameter 
            let query = Object.assign(
                {
    },
                {
     cookie: req.cookies },
                req.query,
                req.body,
                req.files,
            )

            try {
    
                //  Execute the module method , That is, initiate a request for Netease cloud music interface 
                const moduleResponse = await moduleDef.module(query, (...params) => {
    
                    //  Parameter injection client IP
                    const obj = [...params]
                    //  Handle ip, In order to achieve IPv4-IPv6 Interworking ,IPv4 The address will be added before ::ffff:
                    let ip = req.ip
                    if (ip.substr(0, 7) == '::ffff:') {
    
                        ip = ip.substr(7)
                    }
                    obj[3] = {
    
                        ...obj[3],
                        ip,
                    }
                    return request(...obj)
                })
                //  After the request is successful , Get... In the response cookie, And through Set-Cookie Response header to this cookie Set to the front-end browser 
                const cookies = moduleResponse.cookie
                if (Array.isArray(cookies) && cookies.length > 0) {
    
                    if (req.protocol === 'https') {
    
                        //  Remove cross domain requests cookie Of SameSite Limit , This attribute is used to restrict third parties Cookie, So as to reduce the safety risk 
                        res.append(
                            'Set-Cookie',
                            cookies.map((cookie) => {
    
                                return cookie + '; SameSite=None; Secure'
                            }),
                        )
                    } else {
    
                        res.append('Set-Cookie', cookies)
                    }
                }
                //  Reply to the request 
                res.status(moduleResponse.status).send(moduleResponse.body)
            } catch (moduleResponse) {
    
                //  Request failure handling 
                //  No responder , return 404
                if (!moduleResponse.body) {
    
                    res.status(404).send({
    
                        code: 404,
                        data: null,
                        msg: 'Not Found',
                    })
                    return
                }
                // 301 The delegate called the interface that needs to be logged in , But I didn't log in 
                if (moduleResponse.body.code == '301')
                    moduleResponse.body.msg = ' Need to log in '
                res.append('Set-Cookie', moduleResponse.cookie)
                res.status(moduleResponse.status).send(moduleResponse.body)
            }
        })
    }

    return app
}

The logic is clear , Register each module as a route , After receiving the corresponding request , take cookie、 Query parameters 、 The request body is passed to the corresponding module , Then request the interface of Netease cloud music , If the request succeeds , Then deal with the return of Netease cloud music interface cookie, Finally, return all the data to the front end , If the interface fails , Then carry out the corresponding processing .

From the query parameters and request body of the request cookie It may not be easy to understand , because cookie It is usually brought from the request body , This should be done mainly to support Node.js Call inside ：

After the request is successful , If the returned data exists cookie, Then some processing will be carried out , First of all, if it is https Request , Then it will be set SameSite=None; Secure,SameSite yes Cookie A property in , To limit third parties Cookie, So as to reduce the safety risk .Chrome 51 Start adding this attribute , To prevent CSRF Attacks and user tracking , There are three optional values ：strict/lax/none, The default is lax, For example, when the domain name is https://123.com In the page of https://456.com Interface of domain name , By default, in addition to navigating to 123 Of get Except for requests , Other requests will not be carried 123 Domain name cookie, If set to strict More strictly , Not at all cookie, So this project is to facilitate cross domain calls , Set to none, No restrictions , Set to none At the same time, you need to set Secure attribute .

Finally through Set-Cookie The response header will cookie Write to the browser on the front end .

Send a request

Next, let's take a look at the... Used to send the request request Method , This method is /util/request.js file , First, some modules are introduced ：

const encrypt = require('./crypto')
const axios = require('axios')
const PacProxyAgent = require('pac-proxy-agent')
const http = require('http')
const https = require('https')
const tunnel = require('tunnel')
const {
     URLSearchParams, URL } = require('url')
const config = require('../util/config.json')
// ...

Then is the specific method of sending the request createRequest, It's a long way to go , Let's take a slow look at ：

const createRequest = (method, url, data = {
     }, options) => {
    
    return new Promise((resolve, reject) => {
    
        let headers = {
     'User-Agent': chooseUserAgent(options.ua) }
        // ...
        })
}

The function returns one Promise, First, define a request header object , And add the User-Agent head , This header will save the browser type 、 Version number 、 Rendering engine , And the operating system 、 edition 、CPU Type information , The standard format is ：

 Browser logo  ( Operating system identification ;  Encryption level identification ;  Browser language )  Rendering engine logo   Version information 

Needless to say , This header is obviously forged to cheat the server , Let it think that the request is from the browser , Instead of also coming from the server .

By default, there are several dead User-Agent The head is selected randomly ：

const chooseUserAgent = (ua = false) => {
    
    const userAgentList = {
    
        mobile: [
            'Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1',
            'Mozilla/5.0 (Linux; Android 9; PCT-AL10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.64 HuaweiBrowser/10.0.3.311 Mobile Safari/537.36',
            // ...
        ],
        pc: [
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:80.0) Gecko/20100101 Firefox/80.0',
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0',
            // ...
        ],
    }
    let realUserAgentList =
        userAgentList[ua] || userAgentList.mobile.concat(userAgentList.pc)
    return ['mobile', 'pc', false].indexOf(ua) > -1
        ? realUserAgentList[Math.floor(Math.random() * realUserAgentList.length)]
    : ua
}

Continue to look at ：

const createRequest = (method, url, data = {
     }, options) => {
    
    return new Promise((resolve, reject) => {
    
        // ...
        //  If it is post request , Modify encoding format 
        if (method.toUpperCase() === 'POST')
            headers['Content-Type'] = 'application/x-www-form-urlencoded'
        //  forge Referer head 
        if (url.includes('music.163.com'))
            headers['Referer'] = 'https://music.163.com'
        //  Set up ip Head 
        let ip = options.realIP || options.ip || ''
        if (ip) {
    
            headers['X-Real-IP'] = ip
            headers['X-Forwarded-For'] = ip
        }
        // ...
    })
}

Continue to set several header fields ,Axios The default encoding format is json, and POST Requests generally use application/x-www-form-urlencoded Coding format .

Referer The header represents the page where the request is sent url, For example https://123.com Intra page call https://456.com Interface ,Referer The header will be set to https://123.com, This head is usually used for anti-theft chains . So this header is also forged to deceive the server that the request comes from their own page .

Next, we set two ip Head ,realIP The front-end manual transmission is required ：

continue ：

const createRequest = (method, url, data = {
     }, options) => {
    
    return new Promise((resolve, reject) => {
    
        // ...
        //  Set up cookie
        if (typeof options.cookie === 'object') {
    
            if (!options.cookie.MUSIC_U) {
    
                //  tourists 
                if (!options.cookie.MUSIC_A) {
    
                    options.cookie.MUSIC_A = config.anonymous_token
                }
            }
            headers['Cookie'] = Object.keys(options.cookie)
                .map(
                (key) =>
                encodeURIComponent(key) +
                '=' +
                encodeURIComponent(options.cookie[key]),
            )
                .join('; ')
        } else if (options.cookie) {
    
            headers['Cookie'] = options.cookie
        }
        // ...
    })
}

Next set up cookie, There are two types , One is object type , This situation cookie It usually comes from query parameters or request body , The other is string , This is what you ask the head to bring in under normal circumstances .MUSIC_U It should be after login cookie 了 ,MUSIC_A It should be a token, Calling some interfaces without login may report an error , So there will be a tourist token：

continue ：

const createRequest = (method, url, data = {
     }, options) => {
    
    return new Promise((resolve, reject) => {
    
        // ...
        if (options.crypto === 'weapi') {
    
            let csrfToken = (headers['Cookie'] || '').match(/_csrf=([^(;|$)]+)/)
            data.csrf_token = csrfToken ? csrfToken[1] : ''
            data = encrypt.weapi(data)
            url = url.replace(/\w*api/, 'weapi')
        } else if (options.crypto === 'linuxapi') {
    
            data = encrypt.linuxapi({
    
                method: method,
                url: url.replace(/\w*api/, 'api'),
                params: data,
            })
            headers['User-Agent'] =
                'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36'
            url = 'https://music.163.com/api/linux/forward'
        } else if (options.crypto === 'eapi') {
    
            const cookie = options.cookie || {
    }
            const csrfToken = cookie['__csrf'] || ''
            const header = {
    
                osver: cookie.osver, // System version 
                deviceId: cookie.deviceId, //encrypt.base64.encode(imei + '\t02:00:00:00:00:00\t5106025eb79a5247\t70ffbaac7')
                appver: cookie.appver || '8.7.01', // app edition 
                versioncode: cookie.versioncode || '140', // Version number 
                mobilename: cookie.mobilename, // equipment model
                buildver: cookie.buildver || Date.now().toString().substr(0, 10),
                resolution: cookie.resolution || '1920x1080', // Device resolution 
                __csrf: csrfToken,
                os: cookie.os || 'android',
                channel: cookie.channel,
                requestId: `${
      Date.now()}_${
      Math.floor(Math.random() * 1000) .toString() .padStart(4, '0')}`,
            }
            if (cookie.MUSIC_U) header['MUSIC_U'] = cookie.MUSIC_U
            if (cookie.MUSIC_A) header['MUSIC_A'] = cookie.MUSIC_A
            headers['Cookie'] = Object.keys(header)
                .map(
                (key) =>
                encodeURIComponent(key) + '=' + encodeURIComponent(header[key]),
            )
                .join('; ')
            data.header = header
            data = encrypt.eapi(options.url, data)
            url = url.replace(/\w*api/, 'eapi')
        }
        // ...
    })
}

This code will be difficult to understand , The author did not understand , Anyway, roughly speaking, this project uses four types of Netease cloud music interfaces ：weapi、linuxapi、eapi、api, such as ：

https://music.163.com/weapi/vipmall/albumproduct/detail
https://music.163.com/eapi/activate/initProfile
https://music.163.com/api/album/detail/dynamic

Request parameters for each type of interface 、 Encryption methods are different , So it needs to be handled separately ：

such as weapi：

let csrfToken = (headers['Cookie'] || '').match(/_csrf=([^(;|$)]+)/)
data.csrf_token = csrfToken ? csrfToken[1] : ''
data = encrypt.weapi(data)
url = url.replace(/\w*api/, 'weapi')

take cookie Medium _csrf The value is taken out and added to the request data , Then encrypt the data ：

const weapi = (object) => {
    
  const text = JSON.stringify(object)
  const secretKey = crypto
    .randomBytes(16)
    .map((n) => base62.charAt(n % 62).charCodeAt())
  return {
    
    params: aesEncrypt(
      Buffer.from(
        aesEncrypt(Buffer.from(text), 'cbc', presetKey, iv).toString('base64'),
      ),
      'cbc',
      secretKey,
      iv,
    ).toString('base64'),
    encSecKey: rsaEncrypt(secretKey.reverse(), publicKey).toString('hex'),
  }
}

See other encryption algorithms ：crypto.js.

As for how to know these , Or Netease cloud music insiders （ Almost impossible ）, Or reverse it , For example, the interface of the web version , Open console , Send a request , Find the location in the source code , Breaking point , View the request data structure , Read the compressed or confused source code and try it slowly , All in all , Pay tribute to these big guys .

continue ：

const createRequest = (method, url, data = {
     }, options) => {
    
    return new Promise((resolve, reject) => {
    
        // ...
        //  Data structure of response 
        const answer = {
     status: 500, body: {
    }, cookie: [] }
        //  Request configuration 
        let settings = {
    
            method: method,
            url: url,
            headers: headers,
            data: new URLSearchParams(data).toString(),
            httpAgent: new http.Agent({
     keepAlive: true }),
            httpsAgent: new https.Agent({
     keepAlive: true }),
        }
        if (options.crypto === 'eapi') settings.encoding = null
        //  Configure agent 
        if (options.proxy) {
    
            if (options.proxy.indexOf('pac') > -1) {
    
                settings.httpAgent = new PacProxyAgent(options.proxy)
                settings.httpsAgent = new PacProxyAgent(options.proxy)
            } else {
    
                const purl = new URL(options.proxy)
                if (purl.hostname) {
    
                    const agent = tunnel.httpsOverHttp({
    
                        proxy: {
    
                            host: purl.hostname,
                            port: purl.port || 80,
                        },
                    })
                    settings.httpsAgent = agent
                    settings.httpAgent = agent
                    settings.proxy = false
                } else {
    
                    console.error(' Invalid proxy configuration , Don't use agents ')
                }
            }
        } else {
    
            settings.proxy = false
        }
        if (options.crypto === 'eapi') {
    
            settings = {
    
                ...settings,
                responseType: 'arraybuffer',
            }
        }
        // ...
    })
}

Here we mainly define the data structure of the response 、 Defines the requested configuration data , And aiming at eapi Did some special treatment , The most important thing is the related configuration of the agent .

Agent yes Node.js Of HTTP A class in the module , Responsible for managing the http Persistence and reuse of client connections . It maintains a queue of pending requests for a given host and port , Reuse a single socket connection for each request , Until the queue is empty , The socket is either destroyed , Or put it in the pool , In the pool, it will be used again to request to the same host and port , In short, it saves every launch http The time required to recreate the socket when requesting , Increase of efficiency .

pac Refers to agent automatic configuration , In fact, it contains a javascript Function text file , This function will decide whether to connect directly or through a proxy , It's a little easier than writing an agent directly , Of course, you need to configure options.proxy Is the remote address of this file , The format is ：'pac+【pac File address 】+'.pac-proxy-agent The module will provide a http.Agent Realization , It will be based on the specified PAC The proxy file determines which HTTP、HTTPS or SOCKS agent , Or direct connection .

As for why to use tunnel modular , The author searched it and still didn't understand , It may be to solve http The interface of the protocol requests Netease cloud music https The problem of protocol interface failure ？ Friends who know can explain in the comment area ~

Last ：

const createRequest = (method, url, data = {
     }, options) => {
    
    return new Promise((resolve, reject) => {
    
        // ...
        axios(settings)
            .then((res) => {
    
                const body = res.data
                //  Will respond to set-cookie In the header cookie Take out , Save directly to the response object 
                answer.cookie = (res.headers['set-cookie'] || []).map((x) =>
                	x.replace(/\s*Domain=[^(;|$)]+;*/, ''),//  Remove domain name restrictions 
                )
                try {
    
                    // eapi The returned data is also encrypted , Need to decrypt 
                    if (options.crypto === 'eapi') {
    
                        answer.body = JSON.parse(encrypt.decrypt(body).toString())
                    } else {
    
                        answer.body = body
                    }
                    answer.status = answer.body.code || res.status
                    //  Unify these status codes as 200, Both represent success 
                    if (
                        [201, 302, 400, 502, 800, 801, 802, 803].indexOf(answer.body.code) > -1
                    ) {
    
                        //  Special status code 
                        answer.status = 200
                    }
                } catch (e) {
    
                    try {
    
                        answer.body = JSON.parse(body.toString())
                    } catch (err) {
    
                        answer.body = body
                    }
                    answer.status = res.status
                }
                answer.status =
                    100 < answer.status && answer.status < 600 ? answer.status : 400
            	//  Status code 200 On behalf of success , Everything else represents failure 
                if (answer.status === 200) resolve(answer)
                else reject(answer)
            })
            .catch((err) => {
    
                answer.status = 502
                answer.body = {
     code: 502, msg: err }
                reject(answer)
            })
    })
}

Commands are less used now Axios Sent the request , Handled the response cookie, Save to the response object , Convenient for subsequent use , In addition, some status codes are processed , You can see try-catch Use more , As for why , It's estimated that we should try more to know what went wrong , If you are interested, you can try it yourself .

summary

This article through the source point of view to understand NeteaseCloudMusicApi Implementation principle of the project , You can see that the whole process is relatively simple . It's nothing more than a request broker , The difficulty lies in finding these interfaces , And reverse analyze the parameters of each interface , Encryption method , Decryption method . Finally, let me remind you , This project is only for learning , Please do not engage in commercial activities or conduct acts of copyright destruction ~