Selecting elements as the target for screenshots based on CSS selectors does not (currently) allow for the selection of elements based on tag content, or relative to DOM elements selected based on tag content.

However, elements can be selected based on tag content using Javascript. It would be useful to allow for the selection of elements via Javascript as well as CSS.

Alternatively, support a method that can be called from a javascript scraper call that will apply a screen shot to a Javascript selected element.

At the moment, selector based screenshots seem to be focused in _selector_javascript(selectors) by:

https://github.com/simonw/shot-scraper/blob/e40070d9bc5334ad8b2ff9e4e3565263aad545f4/shot_scraper/cli.py#L520

As well as passing --selector(s), s, one approach might be to pass element(s) el returned from a --js-selector script?

The initial prototype of this will involve a YAML file that defines multiple screenshots to be taken, and a GitHub Actions workflow that takes those screenshots.

A repository template that helps users create a repo that takes screenshots of a page.

Create your own template from the repo and it will give you a YAML file that you can then edit - it writes screenshots to the same repository.

This is a bit of an out-there idea: what if you could execute custom JavaScript that returned a result, and then write that result to disk?

You could even skip the screenshot entirely and use this as a generic scraping tool at that point.

Bonus: if it can affect the exit code in some way it could be used as part of a CI flow to test something.

I can use the same tech as https://django-sql-dashboard.datasette.io/en/stable/ - see https://github.com/simonw/django-sql-dashboard/tree/main/docs

• [x] Configure Read the Docs
• [x] Split documentation up into pages
• [x] Add new Contributing page with notes on how docs work (including Cog)
• [x] Vanity domain - probably shot-scraper.datasette.io
• [x] Shrink the README and point it to the new docs
• [x] Ship a release and check that Read the Docs has a "stable" as well as a "latest"

This then made me think that I actually want the same ability for just raw CSS selectors. But what should that look like?

Originally posted by @simonw in https://github.com/simonw/shot-scraper/issues/43#issuecomment-1095338690

A repository template that helps users create a repo that takes screenshots of a page.

Create your own template from the repo and it will give you a YAML file that you can then edit - it writes screenshots to the same repository.

This is a bit of an unconventional need, but I think it's worth exploring.

When taking screenshots for tutorials, I often want to grab an area of the screen that incorporates more than one element - where there's no convenient wrapper element that I can use to get the shot that I want.

Imagine if you could specify multiple selectors and get back a screenshot of the smallest area of the screen that incorporates all of those elements.

The implement would look at the founding box of all of those elements, generate a new box that wraps all of those, inject an absolutely positioned box of that size and take the screenshot of that area.

It could even optionally add some padding to that box before taking the shot.

Got this error when running:

shot-scraper https://lite.datasette.io/ --wait-for 'document.querySelector("h2")' --log-requests - | tee /tmp/datasette-lite.txt

Exception in callback SyncBase._sync.<locals>.callback(<Task finishe...ifier found')>) at /Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_sync_base.py:104
handle: <Handle SyncBase._sync.<locals>.callback(<Task finishe...ifier found')>) at /Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_sync_base.py:104>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 105, in callback
    g_self.switch()
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_browser_context.py", line 122, in <lambda>
    lambda params: self._on_response(
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_browser_context.py", line 397, in _on_response
    page.emit(Page.Events.Response, response)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_base.py", line 113, in emit
    handled = self._call_handlers(event, args, kwargs)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_base.py", line 96, in _call_handlers
    self._emit_run(f, args, kwargs)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_asyncio.py", line 42, in _emit_run
    self.emit('error', exc)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_base.py", line 116, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_base.py", line 86, in _emit_handle_potential_error
    raise error
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_asyncio.py", line 40, in _emit_run
    coro = f(*args, **kwargs)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_impl_to_api_mapping.py", line 88, in wrapper_func
    return handler(
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/shot_scraper/cli.py", line 734, in on_response
    "size": len(response.body()),
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 574, in body
    self._sync("response.body", self._impl_obj.body())
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 111, in _sync
    return task.result()
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_network.py", line 375, in body
    binary = await self._channel.send("body")
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 39, in send
    return await self.inner_send(method, params, False)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 63, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.Error: Protocol error (Network.getResponseBody): No resource with given identifier found

This was logged out a bunch of times, even though the command itself ran to completion.

I think this is likely caused by the new log requests feature from:

• #88

Is there any option to add params to send post to url?

I need to send this key:value pair: "data-raw": "page=21&limit=15"

Or maybe with javascript there is a way to send POST to url?

I tried in javascript doing something like that, but no success:

--javascript "var xhr = new XMLHttpRequest(); xhr.open('POST', <url>, true); xhr.setRequestHeader('Content-Type', 'application/json'); xhr.send(JSON.stringify({'data-raw': 'page=21&limit=15'}));"

Only does the screenshot if the output: file doesn't exist yet.

This could be used to build a very neat GitHub mechanism for creating a portfolio of screenshots of your projects.

I know you use Playwright here @simonw - but I noted this recently: https://blog.cloudflare.com/introducing-workers-browser-rendering-api/

If it's possible to use an alternative rendering engine and offload scraping to a CF Worker, that would be awesome.

Just parking here for your thoughts, and potentially if others are interested. Cloudflare should perhaps be told to use alternative rendering engines too :)

https://discord.com/channels/823971286308356157/1034259126076833873/1034303037914746942 @jefftriplett

I was curious about the multi subcommand and if it made sense to expose screenshot api options (width + height + quality) as cli options. it's not a deal breaker but it felt like it should be a pass-through since I might have a file of 50 urls and I want them to all use the same options. it's slightly more maintainable to keep track of one list of settings and it's nice because I can grab a list for the full width images and then do another for a different size but it's doable without

I like the idea that you can pass options to shot-scraper multi which will be used for YAML items that don't over-ride them.

• option to use system browser
• option to add custom browser command-line arguments
• option to ignore https headers

:books: Documentation preview :books:: https://shot-scraper--91.org.readthedocs.build/en/91/

url to pdf ERROR b'Traceback (most recent call last):\n File "/opt/hostedtoolcache/Python/3.10.4/x64/bin/shot-scraper", line 8, in \n sys.exit(cli())\n File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/click/core.py", line 1130, in call\n return self.main(*args, **kwargs)\n File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/click/core.py", line 1055, in main\n rv = self.invoke(ctx)\n File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/click/core.py", line 1657, in invoke\n return _process_result(sub_ctx.command.invoke(sub_ctx))\n File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/click/core.py", line 1404, in invoke\n return ctx.invoke(self.callback, **ctx.params)\n File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/click/core.py", line 760, in invoke\n return __callback(*args, **kwargs)\n File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packag...

pdfname=url.split('/')[-1]

if not os.path.exists(join(pdfpath,pdfname+'.pdf')):
    cmd='shot-scraper pdf {url} -o {pdfname}.pdf'
    isdone=subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
    out, err = isdone.communicate()

I'm trying to make a screenshot for the page with dropdown menu activated. See example here: https://demoqa.com/menu# ( .nav-menu-container #nav li:nth-child(2) :hover element).

I would appreciate any tips and workflow examples.