API to parse tibia.com content into python objects.

Overview

Tibia.py

An API to parse Tibia.com content into object oriented data.

No fetching is done by this module, you must provide the html content.

GitHub Workflow Status codecov PyPI PyPI - Python Version PyPI - License

Features:

  • Converts data into well-structured Python objects.
  • Type consistent attributes.
  • All objects can be converted to JSON strings.
  • Can be used with any networking library.
  • Support for characters, guilds, houses and worlds, tournaments, forums, etc.

Installing

Install and update using pip

pip install tibia.py

Installing the latest version form GitHub

pip install git+https://github.com/Galarzaa90/tibia.py.git -U

Usage

This library is composed of two parts, parsers and an asynchronous request client.

The asynchronous client (tibiapy.Client) contains methods to obtain information from Tibia.com.

The parsing methods allow you to get Python objects given the html content of a page.

import tibiapy

# Asynchronously
import aiohttp

async def get_character(name):
    url = tibiapy.Character.get_url(name)

    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            content = await resp.text()
    character = tibiapy.Character.from_content(content)
    return character

# Synchronously
import requests

def get_character_sync(name):
    url = tibiapy.Character.get_url(name)
    
    r = requests.get(url)
    content = r.text
    character = tibiapy.Character.from_content(content)
    return character

Documentation

https://tibiapy.readthedocs.io/

Comments
  • DisplayItems attribute name is returning None

    DisplayItems attribute name is returning None

    I was querying the items of a character at the bazaar and the DisplayItem attribute name is returning None

    Here is my code snippet:

    response : TibiaResponse = self.client.loop.run_until_complete(self.client.fetch_auction(character_auction_id, 
            fetch_items=True))
            
    details : AuctionDetails = response.data
    
    items : List[DisplayItem] = details.items.entries
    
    print(items)
    

    The print of the returned list:

    [<DisplayItem name=None count=22 item_id=238>, <DisplayItem name=None count=347 item_id=239>, <DisplayItem name=None count=11 item_id=266>]

    Yesterday everything was working fine, the problem emerged this morning. Maybe the Tibia Bazaar website changed something in their HTML indexing?

    opened by guriosam 6
  • Problem with the comma when there are more than 1000 players online.

    Problem with the comma when there are more than 1000 players online.

    Hello! Cipsoft added a comma in the world list when there are more than 1000 players online.

    File "\site-packages\tibiapy\world.py", line 267, in from_content world._parse_world_info(tables.get("World Information", [])) File "\site-packages\tibiapy\world.py", line 344, in _parse_world_info self.online_count = int(world_info.pop("players_online")) ValueError: invalid literal for int() with base 10: '1,024'

    opened by Pixelao 2
  • tibiapy.HousesSection.from_content is broken

    tibiapy.HousesSection.from_content is broken

    Problem Method tibiapy.HousesSection.from_content(link to code) throws tibiapy.errors.InvalidContent error instead of returning a HousesSection object.

    The reason this unintended behavior happens is because we try to access a Nonetype object inside HouseSection._parse_filters method when we call it house_results._parse_filters(tables["House Search"]) in line 172.

    Quick fix Commenting line 172 solves the issue for me (at least it does for my use case).

    opened by aaronga97 1
  • v2.4.0

    v2.4.0

    Changes

    • Added support for multiple houses per character. Accessible on Character.houses field.
    • Character.house is now deprecated. It will contain the character's first house or None.
    opened by Galarzaa90 1
  • v1.1.0

    v1.1.0

    Changes

    • Parsing Highscores from Tibia.com and TibiaData.
    • Some strings from TibiaData had unpredictable trailing whitespaces, all leading and trailing whitespaces are removed.
    • Added type hints to many variables and methods.
    opened by Galarzaa90 1
  • Parse character position

    Parse character position

    Character's position is not being parsed, for example, Steve has CipSoft Member position. This seems to be the only available position.

    Not to be confused with the position field in Account Information.

    Also, these characters show Cipsoft Member in their status when listed as "Other Characters".

    enhancement 
    opened by Galarzaa90 1
  • Add HouseStatus and Sex enums

    Add HouseStatus and Sex enums

    Testing the waters to maybe improve code. Those do not seem to be used in many places, so they are both good testing subjects.

    Tests kept successful so it seems to be ok. This might have some limitations with json parsing constructors, so let's discuss what you think later :)

    opened by Tschis 1
  • Refactor guild.py and unit tests

    Refactor guild.py and unit tests

    • Refactor guild.py to split different sections of the parsing method. This makes it easier to test and analyze/maintain the code in the future;
    • Add new resource and unit tests for best case scenario on each section;
    opened by Tschis 1
  • Parse character titles and main character information

    Parse character titles and main character information

    Summer update 2019 introduced character titles, as well as a small icon that indicates which is the main character of the account in the other characters section.

    opened by Galarzaa90 0
  • v1.0.0

    v1.0.0

    Changes

    • Added support for TibiaData JSON parsing. To have interoperability between Tibia.com and TibiaData.
    • Added support for parsing Houses, House lists, World and World list
    • Added support for many missing attributes in Character and Guilds.
    • All objects are now serializable to JSON strings.
    opened by Galarzaa90 0
  • Add /tests as a package

    Add /tests as a package

    As seen here, this might help running the tests from the root folder.

    Also added some changes that might facilitate changes into the current test case or new test cases to be added in the future (relative resources path).

    opened by Tschis 0
Releases(v5.5.2)
  • v5.5.2(Sep 3, 2022)

    Changes

    • Fixed Houses section not parsing due to a change in the filters table.
    • Fixed status parameter not generating the correct URL in the houses section.
    Source code(tar.gz)
    Source code(zip)
  • v5.5.1(Aug 2, 2022)

    Changes

    • Adjusted parsing to support the changes related to mobile devices introduced on the day of this release. The following sections were affected:
      • Highscores
      • News
      • Forums
      • Spells
    Source code(tar.gz)
    Source code(zip)
  • v5.5.0(Jul 27, 2022)

  • v5.4.0(Jul 23, 2022)

  • v5.3.0(Jul 23, 2022)

  • v5.2.1(Mar 1, 2022)

  • v5.2.0(Dec 31, 2021)

  • v5.1.0(Sep 16, 2021)

    Changes

    • Added traded attribute to death killers, to indicate that the killer was traded after the death occurred.
    • Properly handle deaths caused by summons of traded characters.
    Source code(tar.gz)
    Source code(zip)
  • v5.0.1.post0(Aug 26, 2021)

  • v5.0.1(Aug 26, 2021)

    Changes

    • Fixed many sections not being parsed correctly due to changes to Tibia.com.
      • Houses list
      • News list
      • Spells section
      • Forums section
    • Fixed character's houses failing to parse due to a bug in the display in Tibia.com.
      • Temporarily disabling this attribute.
    Source code(tar.gz)
    Source code(zip)
  • v5.0.1a1(Aug 24, 2021)

    Prelease to address part of the Tibia.com changes.

    Changes

    • Fixed character's houses failing to parse due to a bug in the display in Tibia.com.
      • Temporarily disabling this attribute.
    • Fixed house list not parsing correctly due to Tibia.com changes
    • Fixed news list not parsing correctly due to Tibia.com changes

    Pending

    • Forums section
    • Spells section
    Source code(tar.gz)
    Source code(zip)
  • v5.0.0(Aug 7, 2021)

    Changes

    • Added parsing for Tibia Drome leaderboards, new Leaderboard class.
      • Auxiliary classes LeaderboardEntry and LeaderboardRotation were added as well.
      • New Client method: fetch_leaderboards.
    • Added parsing for Spells library, new SpellsSection class.
      • Auxiliary classes Spell and SpellEntry.
      • New Client methods: fetch_spell and fetch_spells
    • Fix last page of highscores having 0 as page value.
    • Using the Client class, you can now fetch and parse content from the test version of www.tibia.com when available.
      • Note that if the test website has changes, parsing might not be possible.
      • Internal URL attributes might still point to the regular website.
    • New HousesSection class, including the house filtering attributes.
    • New NewsArchive class, including the news filtering attributes.
    • New GuildsSection class, to replace the lists of GuildEntry
    • Many "ListedObject" classes were renamed to "ObjectEntry", for details check the breaking changes below.
    • Fixed bug with other characters not being parsed.
    • Added traded attribute to LastPost class.
    • Added thread_starter_traded attribute to ThreadEntry

    Breaking Changes:

    • Python 3.7 or higher is now required.
    • House classes no longer have get_list_url and list_from_content methods.
    • Client.fetch_world_houses now returns a HousesSection instance in its data attribute, instead of a list of ListedHouses.
    • ListedHouse.highest_bid attribute now may be None if the house's auction has not yet started.
    • ListedHouse class renamed to HouseEntry.
    • Removed deprecated property AuctionFilters.item.
    • Client.fetch_news_archive, Client.fetch_recent_news now returns an instance of NewsArchive in the data attribute.
    • ListedNews class renamed to NewsEntry.
    • News and NewsEntry no longer have a get_list_url method.
    • ListedBoard class renamed to BoardEntry.
    • ListedThread class renamed to ThreadEntry.
    • ListedAnnouncement class renamed to AnnouncementEntry.
    • ListedWorld class renamed to WorldEntry.
    • ListedAuction class renamed to AuctionEntry.
    • AuctionDetails class renamed to Auction.
    • ListedGuild class renamed to GuildEntry.
    • ListedTournament class renamed to TournamentEntry.
    • Creature class renamed to CreatureEntry.
    • CreatureDetail class renamed to Creature.
    • Guild and GuildEntry class no longer have a get_list_url method.
    • Renamed begin_date parameter to start_date in fetch_news_archive.
    • Renamed race attribute of CreatureEntry and Creature to identifier, method parameters renamed as well.
    • CreaturesSection.from_boosted_creature_header renamed to CreaturesSection.boosted_creature_from_header.
    Source code(tar.gz)
    Source code(zip)
  • v4.1.7(Jun 30, 2021)

  • v4.1.6(Jun 28, 2021)

    Changes

    • Fixed worlds not being parsed correctly again due to tournament worlds order changing. After this fix, the order should not matter anymore.
    Source code(tar.gz)
    Source code(zip)
  • v4.1.5(Jun 26, 2021)

  • v4.1.4(Jun 17, 2021)

  • v4.1.3(May 12, 2021)

  • v4.1.2(Apr 27, 2021)

  • v4.1.1(Apr 19, 2021)

  • v4.1.0(Mar 30, 2021)

    Changes

    • Added prey_wildcards attribute to AuctionDetails.
    • Added filters parameter to CharacterBazaar.get_auctions_history_url and Client.fetch_auction_history.
    Source code(tar.gz)
    Source code(zip)
  • v4.0.0(Mar 11, 2021)

    Changes

    • Breaking change: Removed BoostedCreature class, replaced by Creature class.
      • Attributes should be compatible, image_url is a property of Creature, calculated from its race attribute.
    • Added parsing and fetching for the Creature library section.
      • Added CreatureSection, Creature, and CreatureDetail classes.
    • Added traded attribute to ForumAuthor. Indicates if the author was a traded character.
      • Previously, it would mark the author as a deleted character and its name would include (traded).
    • Fixed a bug with ForumBoards not parsing due to the cookies dialog that was added.
    • Added battleye_type attribute to ListedWorld and World classes. Indicates the type of BattlEye protection the world has.
      • battleye_protected is now a property instead of an attribute.
    • Added YELLOW and GREEN aliases to all BattlEye related enums.
    • Fixed wrong timezone being used for forum related dates.
    Source code(tar.gz)
    Source code(zip)
  • v3.7.1(Feb 16, 2021)

  • v.3.7.0(Feb 9, 2021)

    Changes

    • Parse familiars from auctions
    • Updated the way tooltips in auctions are parsed, the format changed, resulting in the previous code not working anymore.
    • Results count in bazaar pages are now properly parsed when there are comma thousand separators.
    • Item amounts are now more accurate instead of being based from their indicator (which was grouping them in thousands)
    Source code(tar.gz)
    Source code(zip)
  • v3.6.5(Jan 28, 2021)

  • v3.6.4(Jan 26, 2021)

  • v3.6.3(Jan 15, 2021)

  • v3.6.2(Jan 2, 2021)

  • v3.6.1(Dec 28, 2020)

  • v3.6.0(Dec 12, 2020)

    Changes

    • Added support for the new filtering options in Highscores
      • Added battleye_filter and pvp_types_filter attributes.
    • Added get_page_url() instance method to Highscores class.
    • Added previous_page_url and next_page_url properties.
    Source code(tar.gz)
    Source code(zip)
  • v3.5.7(Dec 4, 2020)

Owner
Allan Galarza
Mechatronic Engineer
Allan Galarza
Automatically scrapes all menu items from the Taco Bell website

Automatically scrapes all menu items from the Taco Bell website. Returns as PANDAS dataframe.

Sasha 2 Jan 15, 2022
A python module to parse the Open Graph Protocol

OpenGraph is a module of python for parsing the Open Graph Protocol, you can read more about the specification at http://ogp.me/ Installation $ pip in

Erik Rivera 213 Nov 12, 2022
This program will help you to properly scrape all data from a specific website

This program will help you to properly scrape all data from a specific website

MD. MINHAZ 0 May 15, 2022
A dead simple crawler to get books information from Douban.

Introduction A dead simple crawler to get books information from Douban. Pre-requesites Python 3 Install dependencies from requirements.txt (Optional)

Yun Wang 1 Jan 10, 2022
HappyScrapper - Google news web scrapper with python

HappyScrapper ~ Google news web scrapper INSTALLATION ♦ Clone the repository ♦ O

Jhon Aguiar 0 Nov 07, 2022
Audio media crawler for lbry.

Audio media crawler for lbry. Requirements Python 3.8 Poetry 1.1.7 Elasticsearch 7.14.0 Lbry-sdk 0.99.0 Development This project uses poetry as a depe

Hound.fm 4 Dec 03, 2022
Docker containerized Python Flask API that uses selenium to scrape and interact with websites

Docker containerized Python Flask API that uses selenium to scrape and interact with websites

Christian Gracia 0 Jan 22, 2022
Scrapes all articles and their headlines from theonion.com

The Onion Article Scraper Scrapes all articles and their headlines from the satirical news website https://www.theonion.com Also see Clickhole Article

0 Nov 17, 2021
🕷 Phone Crawler with multi-thread functionality

Phone Crawler: Phone Crawler with multi-thread functionality Disclaimer: I'm not responsible for any illegal/misuse actions, this program was made for

Kmuv1t 3 Feb 10, 2022
This program scrapes information and images for movies and TV shows.

Media-WebScraper This program scrapes information and images for movies and TV shows. Summary For more information on the program, read the WebScrape_

1 Dec 05, 2021
A Python package that scrapes Google News article data while remaining undetected by Google.

A Python package that scrapes Google News article data while remaining undetected by Google. Our scraper can scrape page data up until the last page and never trigger a CAPTCHA (download stats: https

Geminid Systems, Inc 6 Aug 10, 2022
Grab the changelog from releases on Github

release-notes-scraper This simple script can be used to grab the release notes for projects from github that do not keep a CHANGELOG, but publish thei

Dan Čermák 4 Apr 01, 2022
An application that on a given url, crowls a web page and gets all words, sorts and counts them.

Web-Scrapping-1 An application that on a given url, crowls a web page and gets all words, sorts and counts them. Installation Using the package manage

adriano atambo 1 Jan 16, 2022
Scrapes mcc-mnc.com and outputs 3 files with the data (JSON, CSV & XLSX)

mcc-mnc.com-webscraper Scrapes mcc-mnc.com and outputs 3 files with the data (JSON, CSV & XLSX) A Python script for web scraping mcc-mnc.com Link: mcc

Anton Ivarsson 1 Nov 07, 2021
A simple reddit scraper to get memes (only images) from r/ProgrammerHumor.

memey A simple reddit scraper to get memes (only images) from r/ProgrammerHumor. Note Only works if you have firefox installed (yet). Instructions foo

2 Nov 16, 2021
A python tool to scrape NFT's off of OpenSea

Right Click Bot A script to download NFT PNG's from OpenSea. All the NFT's you could ever want, no blockchain, for free. Usage Must Use Python 3! Auto

15 Jul 16, 2022
This script is intended to crawl license information of repositories through the GitHub API.

GithubLicenseCrawler This script is intended to crawl license information of repositories through the GitHub API. Taking a csv file with requirements.

schutera 4 Oct 25, 2022
淘宝茅台抢购最新优化版本,淘宝茅台秒杀,优化了茅台抢购线程队列

淘宝茅台抢购最新优化版本,淘宝茅台秒杀,优化了茅台抢购线程队列

MaoTai 118 Dec 16, 2022
A Scrapper with python

Scrapper-en-python Scrapper des données signifie récuperer des données pour les traiter ou les analyser. En python, il y'a 2 grands moyens de scrapper

Lun4rIum 1 Dec 05, 2021
This is a web crawler that works on employ email data by gmane.org and visualizes it in different ways.

crawler_to_visual_gmane Analyzing an EMAIL Archive from gmane and vizualizing the data using the D3 JavaScript library. This is a set of tools that al

Saim Zafar 1 Dec 20, 2021