Climbing 10000 NASA pictures about Mars exploration, I found a secret

Preface

lately , I use crawler technology , Crawling NASA , That's what you often see in movies NASA, Pictures of Mars exploration , Yes 10000 Zhang Ba .

Mm-hmm , Trifle thing , Trifle thing .

When it's done , A little excited , Hence the article , There will be the following ：

1. Why do I crawl NASA Pictures of the
2. How do I crawl NASA The image （ Hyperdetail ）
3. What do I get ( HD map )
4. What secret did I discover （ Super powerful ）

Why do I climb NASA Pictures of the

I've passed 35 了 , Shivering when it was opened .

Every day I think about what to do if I lose my job , Think about playing we media , I'll talk to you everyday . Vernacular is a historical mystery , The mysteries of the universe , So I got my eye on NASA.

NASA There are all kinds of space exploration missions , And there are related articles , interview , picture , The video is public , This is a rare resource pool .

How do I crawl NASA The image （ Hyperdetail ）

NASA Our website is publicly accessible , The address is

https://www.nasa.gov/

Wechat pictures _20210630130517.png

After opening , It looks like this , You can see all kinds of content . There's also a search box in the upper right corner , We type in Mars That is to say Mars

Wait a moment , Exhibition shows and Mars All kinds of related content , One of them Mars Exploration That is to say Mars exploration

After the light , It's a new page , And then find Images picture , It's the target page we crawled

https://www.nasa.gov/mission_pages/mars/images/index.html

Wechat pictures _20210630130632.png

Page drop down , You'll see a big button , It reads MORE IMAGES, Click to have a try and you'll find ：

Content of page , It's not a page loaded directly , But through api After the request , Rendered asynchronously

F12, Open browser developer mode , Repeat the steps , Observe the request information , I find that there will be the following situations

Wechat pictures _20210630123102.png

Looks like this url The address is very important , Let's look at his request address first ：

https://www.nasa.gov/api/2/ubernode/_search?size=24&from=24&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi

Pay attention to the parameters

size=24&from=24

Obviously ,size It's the number of images per request ,from After testing , Is the initial location of the query , We can change it to get other content

Let's look at its return message ：

{
	"took": 3,
	"timed_out": false,
	"_shards": {
		"total": 5,
		"successful": 5,
		"skipped": 0,
		"failed": 0
	},
	"hits": {
		"total": 659,
		"max_score": null,
		"hits": [{
			"_index": "nasa-public",
			"_type": "ubernode",
			"_id": "450040",
			"_score": null,
			"_source": {
				"image-feature-caption": "Mars 2020 rover underwent an eye exam after several cameras were installed on the rover. ",
				"topics": ["3140", "3152"],
				"nid": "450040",
				"title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision",
				"type": "ubernode",
				"uri": "/image-feature/jpl/nasa-optometrists-verify-mars-2020-rovers-2020-vision",
				"collections": ["4525", "5246"],
				"link-or-attachment": "link",
				"missions": ["6336"],
				"primary-tag": "6336",
				"cardfeed-title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision",
				"promo-date-time": "2019-08-05T17:49:00-04:00",
				"secondary-tag": "3140",
				"master-image": {
					"fid": "603128",
					"alt": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ",
					"width": "1600",
					"id": "603128",
					"title": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ",
					"uri": "public://thumbnails/image/pia23314-16.jpg",
					"height": "900"
				},
				"ubernode-type": "image"
			},
			"sort": [1565041740000]
		}, {
			"_index": "nasa-public",
			"_type": "ubernode",
			"_id": "433172",
			"_score": null,
			"_source": {
				"image-feature-caption": "NASA still hasn't heard from the Opportunity rover, but at least we can see it again.",
				"topics": ["3152"],
				"nid": "433172",
				"title": "Opportunity Emerges in a Dusty Picture",
				"type": "ubernode",
				"uri": "/image-feature/opportunity-emerges-in-a-dusty-picture",
				"collections": ["7628"],
				"link-or-attachment": "link",
				"missions": ["3639"],
				"primary-tag": "3152",
				"cardfeed-title": "Opportunity Emerges in a Dusty Picture",
				"promo-date-time": "2018-09-26T12:39:00-04:00",
				"secondary-tag": "7628",
				"master-image": {
					"fid": "584263",
					"alt": "NASA's Opportunity rover appears as a blip in the center of this square",
					"width": "1400",
					"id": "584263",
					"title": "NASA's Opportunity rover appears as a blip in the center of this square",
					"uri": "public://thumbnails/image/pia22549-16.jpg",
					"height": "788"
				},
				"ubernode-type": "image"
			},
			"sort": [1537979940000]
		}]
	}
}

above json The content is too long , I deleted some of the duplicate , actually hits This array , It's also 24 individual , It's the same number of images that are displayed on the page . It can be concluded that , Information on the page , It's from this array .

And then we found that ,master-image Under the field , It's the information we need , Include Picture address , Picture size , Picture title .

Here's the code , Assembly request URL, Get content , Download the pictures Three steps

I use the Dart Language , You are at liberty

import 'dart:convert';
import 'package:dio/dio.dart';

main() async {
  //  The number of pages is fixed 24 individual , Just change the initial value 
  for (int from = 0; from < 24 * 100; from = from + 24) {
    await getPage(from);
  }
}

// Get information on every page and download 
Future<void> getPage(int from) async {
  String url = 'https://www.nasa.gov/api/2/ubernode/_search?size=24&from=' +
      from.toString() +
      '&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi';

  // Get content 
  var res = await Dio().get(url);
  var map = jsonDecode(res.toString());

  (map['hits']['hits'] as List<dynamic>).forEach((element) async {
    Uri fileUri = Uri.parse(getUri(element));
    String savePath = getSavePath(element);

    await Dio().downloadUri(fileUri, savePath);
    print(' Downloaded : ' + savePath);
  });
}

// Get the picture download address 
String getUri(dynamic element) {
  String uri = element['_source']['master-image']['uri'].toString();

  uri = uri.replaceAll('public://',
      'https://www.nasa.gov/sites/default/files/styles/full_width_feature/public/');

  return uri;
}

// process information , And return to the picture save address 
String getSavePath(dynamic element) {
  String id = element['_id'];
  String fid = element['_source']['master-image']['fid'].toString();
  String title = element['_source']['master-image']['title'].toString();
  String uri = element['_source']['master-image']['uri'].toString();

  String savePath =
      id + '_' + fid + '_' + title.trim() + '.' + uri.split('.').last;
  savePath = savePath.replaceAll('/', '');
  savePath = savePath.replaceAll('\\', '');
  savePath = savePath.replaceAll('"', '');
  savePath = 'images/' + savePath;

  return savePath;
}

The above code , It's still very simple , Experienced students should understand it as soon as they read it .

Let's go .

 Downloaded : images/470436_643588_This is the third color image taken by NASA’s Ingenuity helicopter.jpg
 Downloaded : images/470435_643587_This is the second color image taken by NASA’s Ingenuity helicopter.jpg
 Downloaded : images/468546_639327_This is the first high-resolution, color image to be sent back by the Hazard Cameras (Hazcams).jpg
 Downloaded : images/452007_605784_Danielson Crater on Mars.jpg
 Downloaded : images/458478_615132_Gullies on Mars.jpg
 Downloaded : images/469416_641582_A field of sand dunes occupies this frosty 5-kilometer diameter crater in the high-latitudes of the northern plains of Mars..jpeg
 Downloaded : images/458075_614251_Mars 2020 With Sample Tubes (Artist's Concept).jpg
 Downloaded : images/470381_643473_CME.jpg
 Downloaded : images/458813_615896_Mars.jpg
 Downloaded : images/467026_635309_Illustration of NASA’s Perseverance rover begins its descent through the Martian atmosphere.jpg
 Downloaded : images/470438_643591_This black and white image was taken by NASA’s Ingenuity helicopter during its third flight on April 25, 2021.jpg
 Downloaded : images/465488_631398_Cliffs in Ancient Ice on Mars.jpg
 Downloaded : images/463659_626874_Avalanche on Mars.jpg
 Downloaded : images/470251_643164_This image from NASA’s Perseverance rover shows the agency’s Ingenuity Mars Helicopter right after it successfully completed a high-speed spin-up test..jpeg
 Downloaded : images/468636_639726_Mars' Jezero Crater.jpg

What do I get

These pictures

Wechat pictures _20210630121020.png

And these

Wechat pictures _20210630131701.png

picture , All the picture titles are available , It's been a month. I guess .

What secret did I discover

429059_580437_two images of mars, one clear, one fuzzy.png

This picture , My favorite . One so clear , One so turbid , Why is that ？ Martian crevice generators ？

ok , The real secret is ：

NASA There is no anti collection on our website , Try it if you don't believe it ...