Having version 1.2.1 it is not possible to launch trafilatura extraction in the async task like celery. https://github.com/adbar/trafilatura/blob/1bb5fee6a4812e53b6597053c25efde995174d79/trafilatura/core.py#L982 It would be better to have HAS_SIGNAL as config variable, and not hardcoded value

celery_1      |     text = trafilatura.extract(
celery_1      |   File "/usr/local/lib/python3.8/site-packages/trafilatura/core.py", line 982, in extract
celery_1      |     signal(SIGALRM, timeout_handler)
celery_1      |   File "/usr/local/lib/python3.8/signal.py", line 47, in signal
celery_1      |     handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
celery_1      | ValueError: signal only works in main thread

Hello,

Thanks for your beautiful and powerful project, I try to test some websites with trafilatura 0.6.0 in Python 3.8.

My test:

import trafilatura
from trafilatura.core import bare_extraction

downloaded = trafilatura.fetch_url('https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/')

result = bare_extraction(downloaded, include_formatting=False, with_metadata=True)

print(result)

The results: ({'title': None, 'author': None, 'url': None, 'hostname': None, 'description': None, 'sitename': None, 'date': None, 'categories': None, 'tags': None, 'fingerprint': None, 'id': None}, 'Leader spotlight: Erin Spiceland Every March we recognize the women who have shaped history—and now, we’re taking a look forward. From driving software development in large companies to maintaining thriving open source communities, we’re spending Women’s History Month with women leaders who are making history every day in the tech community. Erin Spiceland is a Software Engineer for SpaceX. Born and raised in rural south Georgia, she is a Choctaw and Chickasaw mother of two now living in downtown Los Angeles. Erin didn’t finish college—she’s a predominantly self-taught software engineer. In her spare time, she makes handmade Native American beadwork and regalia and attends powwows. How would you summarize your career (so far) in a single sentence? My career has been a winding road through periods of stimulation and health as well as periods of personal misery. During it all, I’ve learned a variety of programming languages and technologies while working on a diverse array of products and services. I’m a domestic abuse survivor and a Choctaw bisexual polyamorous woman. I’m so proud of myself that I made it this far considering where I came from. What was your first job in tech like? In 2007, I had a three-year-old daughter and I was trying to finish my computer science degree one class at a time, all while keeping my house and family running smoothly. I found the math classes exciting and quickly finished my math minor, leaving only computer science classes. I was looking at about five years before I would graduate. Then, my husband at the time recommended me for an entry software developer position at a telecom and digital communications company. When faced with the choice between an expensive computer science degree and getting paid to do what I loved, I dropped out of college and accepted the job. I was hired to work on internal tooling, and eventually, products. I did a lot of development on product front-ends, embedded network devices, and a distributed platform-as-a-service. I learned Java/JSP, Python, JavaScript/CSS, Node.js, as well as MySQL, PostgreSQL, and distributed systems architecture. It was an intense experience that required a lot of self-teaching, asking others for help, and daycare, but it set me up for my later successes. What does leadership mean to you in your current role? “Leadership is about enabling those below, above, and around you to be at their healthiest and most effective so that all of you can accurately understand your surroundings, make effective plans and goals for the future, and achieve those goals.” I appreciate and admire technical, effective leaders who care for their reports as humans, not as lines on a burndown chart, and forego heavy-handed direction in favor of communication and mutual dialogue. I think it’s as important for a leader to concern herself with her coworkers’ personal well-being as it is for her to direct their performance. What’s the biggest career risk you’ve ever taken? What did you learn from that experience? Last year I took a pay cut to move from a safe, easy job where I had security to work in a language I hadn’t seen in years and with systems more complicated than anything I’d worked with before. I moved from a place where I had a huge four bedroom house to a studio apartment that was twice the price. I moved away from my children, of who I share custody with my ex-husband. We fly across the U.S. to see each other now. I miss my children every day. However, I get to be a wonderful role model for them. “I get to show my children that a Native woman who grew up in poverty, lost her mother and her culture, and who didn’t finish college can learn, grow, and build whatever career and life she wants.” What are you looking forward to next? I can’t wait to wake up every day with my partner who loves me so much. I’m looking forward to showing my children exactly how far they can go. I’m excited to keep exploring Los Angeles. “I expect to learn so much more about software and about life, and I want to experience everything.” Want to know more about Erin Spiceland? Follow them on GitHub or Twitter. Want to learn more about featured leaders for Women’s History Month? Read about: Laura Frank Tacho, Director of Engineering at CloudBees Rachel White, Developer Experience Lead at American Express Kathy Pham, Computer Scientist and Product Leader at Mozilla and Harvard Heidy Khlaaf, Research Consultant at Adelard LLP Check back in soon—we’ll be adding new interviews weekly throughout March.', <Element body at 0x10680a280>, <Element body at 0x1067af080>)

So, no metadata return.

Also, I added a xpath in the metaxpaths.py and rebuild your code. I'm sure that //div[contains(@class, "post__categories")]//li//a will be match with a category in the url https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/. But no category is returned.

categories_xpaths = [
    """//div[starts-with(@class, 'post-info') or starts-with(@class, 'postinfo') or
    starts-with(@class, 'post-meta') or starts-with(@class, 'postmeta') or
    starts-with(@class, 'meta') or starts-with(@class, 'entry-meta') or starts-with(@class, 'entry-info') or
    starts-with(@class, 'entry-utility') or starts-with(@id, 'postpath')]//a""",
    "//p[starts-with(@class, 'postmeta') or starts-with(@class, 'entry-categories') or @class='postinfo' or @id='filedunder']//a",
    "//footer[starts-with(@class, 'entry-meta') or starts-with(@class, 'entry-footer') or starts-with(@class, 'post-info')]//a",
    '//*[(self::li or self::span)][@class="post-category" or starts-with(@class, "post__categories") or @class="postcategory" or @class="entry-category"]//a',
    '//header[@class="entry-header"]//a',
    '//div[@class="row" or @class="tags"]//a',
    '//div[contains(@class, "post__categories")]//li//a',
]

Another question is that could I get content of article including html format (no clean tags in content)?

Please help me, thanks for your support!

We shouldnt believe on schema person

agenda Current: "author": "Sandy Cheu", Should be: "author": "Stephen Teulan; Nikita Weikhardt",

aged Current: "author":"Consumers", Should be: "author": "Liz Alderslade",

meta remove single names cath Current: "author": null, Should be: "author": "Rebecca",

echo Current: "author": null, Should be: "author": "Katie",

The current repo should work well? I have removed several things that are unused and fixed a tiny bug that affects the accuracy. I have added to the git ignore so that the branch should now get quite clean as well XD

When using include_formatting for plain text, I'm not seeing any formatting (bold, italics, etc..). The term I'm using supports this. Is this by design or a bug? I tried both the standalone version and using it as a library with trafilatura.extract(downloaded, include_formatting=True).

I. Test file

from time import time

import requests
from trafilatura import extract


if __name__ == '__main__':
    urls = ["https://en.wikipedia.org/wiki/List_of_Hindi_songs_recorded_by_Asha_Bhosle",
            "https://en.wikipedia.org/wiki/2022_in_video_games",
            "https://en.wikipedia.org/wiki/COVID-19_pandemic_in_Kuwait",
            "https://en.wikipedia.org/wiki/Presidency_of_Rodrigo_Duterte",
            "https://en.wikipedia.org/wiki/List_of_2021%E2%80%9322_NBA_season_transactions",
            "https://en.wikipedia.org/wiki/2022_in_sports",
            "https://en.wikipedia.org/wiki/Firefox_version_history",
            "https://en.wikipedia.org/wiki/List_of_common_misconceptions",
            "https://en.wikipedia.org/wiki/Same-sex_union_legislation",
            "https://en.wikipedia.org/wiki/Presidency_of_Donald_Trump",]

    cum_time = 0
    for url in urls:        
        resp = requests.get(url)
        t0 = time()
        result = extract(resp.text)
        cum_time = cum_time + time() - t0
    print(cum_time)

II. Test pprofile

kernprof -lv test2.py

before

Total time: 0.544693 s
File: /trafilatura-master/trafilatura/utils.py
Function: remove_control_characters at line 221

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   221                                           @profile
   222                                           def remove_control_characters(string):
   223                                               '''Prevent non-printable and XML invalid character errors'''
   224     25998     544693.0     21.0    100.0      return ''.join([c for c in string if c.isprintable() or c.isspace()])

after

Total time: 0.169241 s
File: /trafilatura-master/trafilatura/utils.py
Function: remove_control_characters at line 227

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   227                                           @profile
   228                                           def remove_control_characters(string):
   229                                               '''Prevent non-printable and XML invalid character errors'''
   230     25998     169241.0      6.5    100.0      return ''.join(filter(is_printable_or_space, string))

III. Test vprof

vprof -c -h test2.py

before

after

In this correction:

• added 'submitted-by' and 'username' tags to xpath
• the maximum size of the author's name has been increased.
• regex has been added to remove emoji from author names often found on sites like buzzfeed
• added a regex to minify json before running the other regex, was having trouble fetching authors when json formatted.
• added a regex to remove json items like images and organization before searching the author
• reorganized the extract_json function as it was overwriting meta tags with none when no json was found

qsr Before this fix: "author": null After this fix: "author": "Kevin Santos"

perthnow Before this fix: "author": "NCA NewsWire" After this fix: "author": "Finn McHugh"

buzzfeed Before this fix: "author": "Hameda Nafiz BuzzFeed Staff" After this fix: "author": "Hameda Nafiz"

buzzfeed Before this fix: "author": "Olivia ❤️" After this fix: "author": "Olivia Community Contributor"

build Before this fix: "author": null After this fix: "author": "Thoams Lane"

hunterandbligh Before this fix: none After this fix: "author": "REBECCA MAGRO"

abc - 'data-component' Before this fix: "author": null After this fix: "author": "Charlotte Gore"

proactiveinvestors Before this fix: "author": null After this fix: "author": "Calum Muirhead"

banking Before this fix: "author": "Sarah Harman Jul" After this fix: "author": "Sarah Harman"

hcamag Before this fix: "author": "Sarah Harman Jul" After this fix: "author": "Mark Rosanes"

spacedaily and + 9 sites Before this fix: "author": null After this fix: "author": "Lucie Aubourg"

first Before this fix:"author": "Nick Griffin", After this fix: "author": "Stan Shamu",

racing Before this fix:"author": "Ben Sporle - @bensporle; Ben Sporle", After this fix: "author": "Ben Sporle",

ajn Before this fix:"author": "RABBI GARY ROBUCK July", After this fix: "author": "RABBI GARY ROBUCK",

ESPN it is not totally fix, but it is better Before this fix: "author": "Andrew Mcglashandeputy Editor, Espncricinfo", After this fix: "author": "Andrew McGlashan Deputy editor; ESPNcricinfo",

Probono it is not totally fix, but it is better Before this fix: "author": null, After this fix: "author": "Luke Michael; Journalist; @Luke_Michael",

If readbility fallback is activated, the Trafilatura library redirects stderr to /dev/null upon every call: https://github.com/adbar/trafilatura/blob/a56fb3e041175df38a32b1c5ef2e9c7888eeb7a6/trafilatura/external.py#L63

Within programs involving other libraries, this causes a host of side effects. E.g., generating a chart with seaborn imports ipython (a dependency of seaborn) which pre-checks upon initialization stdin, stdout and stderr and crashes because stderr is /dev/null. I have other side effects as well in other libraries, including disappearing logs (eg when logs settings are modified after calls to Trafilatura).

This redirection seems to have been necessary to prevent the readibility library to print out messages to stderr. A cursory reading of the current version of readibility seems to indicate it doesn't do that, it only emits proper logs.

Consequently, this redirect may be removed (to be tested).

I'm not quite sure where to begin with this, it's a strange one. In a real world scenario I tried switching from Goose3 to Trafilatura. I'm processing html extractions in parallel with dask. After switching to trafilatura, I noticed a 30% slowdown. I ended up writing my own evaluation library to verify the results.

Results from running in parallel: ┏━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Library ┃ Accuracy ┃ Precision ┃ Recall ┃ FScore ┃ Mean Similarity ┃ Items/sec ┃ ┡━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ goose3 │ 0.9678 │ 0.8561 │ 0.9547 │ 0.9027 │ 0.8343 │ 383.4737 │ │ trafilatura │ 0.9124 │ 0.9485 │ 0.908 │ 0.9278 │ 0.8567 │ 361.3232 │ └─────────────┴──────────┴───────────┴────────┴────────┴─────────────────┴───────────┘

Results from running sequentially: ┏━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Library ┃ Accuracy ┃ Precision ┃ Recall ┃ FScore ┃ Mean Similarity ┃ Items/sec ┃ ┡━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ goose3 │ 0.9678 │ 0.8561 │ 0.9547 │ 0.9027 │ 0.8343 │ 9.7953 │ │ trafilatura │ 0.9124 │ 0.9485 │ 0.908 │ 0.9278 │ 0.8567 │ 23.0045 │ └─────────────┴──────────┴───────────┴────────┴────────┴─────────────────┴───────────┘

Note: the dataset evaluated in from scrapinghub/article-extraction-benchmark tool. The only portion of the code that runs in parallel for the bench marks is the extraction. Only the extraction is timed for calculating items/sec.

In summary: trafilatura is marginally slower than Goose3 in parallel. However sequentially it is twice as fast as Goose3.

I'm not sure where to begin with this. It can be difficult to profile parallel processing. It may be related to some of the memory leak issues reported with trafilutura, although it appears those have been resolved. Or the caching, I haven't looked into how that functions.

I will work on publishing my benchmarking tool this afternoon.

This fixes #85 (and #159).

It involved a bit of a refactor of the extract_content function, but the basic idea is that it looks through all of the children in the subtree returned from tree.xpath(expr), not just stopping at the first child like before. Beyond that, it pulls out the logic that checks whether the BODY_XPATH expression matched in the current loop iteration has found a useful subtree, to make it a little more readable, and only performs the final cleanup and look-elsewhere logic at the very end.

So essentially, on finding a subtree whose first node is valid, we proceeded to consider all of the remaining nodes in that subtree.

This seems to work great, although I haven't run it through the automated tests. (I had trouble running the url tests.)

Let me know what you think. Happy to talk through anything, and if/when this seems good to you, I'll clean it up (print statements, code style, etc.).

Thanks!

I'm not quite sure what's wrong with images but here is reproducer:

$ curl https://en.wikipedia.org/wiki/Tribe > /tmp/tribe.html
$ python
Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> html_wiki_tribe = open('/tmp/tribe.html').read()
... text = trafilatura.extract(
...     html_wiki_tribe,
...     include_images=True
... )
~/anaconda3/lib/python3.7/site-packages/trafilatura/xml.py in xmltotxt(xmloutput, include_formatting, include_links)
    272             LOGGER.debug('unexpected element: %s', element.tag)
    273             returnlist.extend([textelement, ' '])
--> 274     return sanitize(''.join(returnlist))
    275 
    276 

TypeError: sequence item 6: expected str instance, NoneType found

UPD Looks like this could help:

Most os sites add a suffix like:

• My article title | My Site Name
• My article title - My Site Name

There is no need the sitename within the article title

Common separators are: - | – — • · ‹ › ⁄ « » < > : * ⋆ ~

Some sites use html entities for this, like: —

Possibility to remove unnecessary html elements before starting the extraction process.

There are often some elements within the extracted text that are not article content.

Titles should by default not come inside the extracted text, or there should be an option to remove them (maybe this requires another issue)

Something like:

unwanted = [
  'iframe',
  'button',
  'figcaption',
  'caption',
  'form',
  'aside',
  'script',
  'style',
  'ins',
  'link',
  'header',
  'footer',
  '#comments',
  'nav',
  '.post-comments',
  '.post-tags',
  '.wp-block-embed',
  '.wp-caption-text',
  'svg',
  '[class^=ads]',
  '[class*=ads-]',
  '[style="display:none"]',
  '[style*="display:none"]',
  '[style*="display: none"]',
  '[itemprop*="description"]',
  '.push-web-notification',
  '.mc-column.entities',
  '.newsletter-component',
  '.post-subject',
  '.post-info',
  '.addthis_tool',
  '.pt-cv-wrapper'
]

article = trafilatura.bare_extraction(document,
        unwanted_elements=unwanted
        include_comments=False, include_tables=False,
        favor_precision=True, favor_recall=True,
        no_fallback=True, target_language=None,
        date_extraction_params={'extensive_search': True, 'original_date': True, 'outputformat': "%Y-%m-%dT%H:%M:%S%z"},
        config=config)

Sometimes an image is not included in text body and we can extract by some SEO TAGS

Issue: https://github.com/adbar/trafilatura/issues/281

Unfortunately I didn't have time to create the tests

Sometimes an image is not included in text body and we can extract by some SEO TAGS like some article parsers do (https://github.com/extractus/article-extractor/blob/main/src/utils/extractMetaData.js)

Here some metatags:

'image'
'og:image'
'og:image:url'
'og:image:secure_url'
'twitter:image'
'twitter:image:src'

Not able to fetch image tags Not able to fetch iframe tags. From command prompt in windows machine

trafilatura --sitemap "https://www.lyricspulp.com/" --list > linklist.txt trafilatura --sitemap homepage --list > linklist.txt trafilatura -i linklist.txt --xml -o outputfile.txt trafilatura -i linklist.txt --formatting --links --images --no-comments --xml -o outputfile.txt