Every web site provides APIs.
Toapi is a clever, simple and fast flask library that enable any website to provide API services. For many occasions, websites do not provide API services for you to download data from them. You have to crawl some data, store them and build an API service, and eventually, you get your data after a tough struggle. And that's not an end, for you might have to update them regularly.
Toapi turns these matters into a piece of cake. All you need to do is to define the data you want, and you've made it. The process is fully automated, and data can be accessed through API in seconds!
- Documentation: http://www.toapi.org
- Awesome: https://github.com/toapi/awesome-toapi
- Organization: https://github.com/toapi
from toapi import XPath, Item, Api
from toapi import Settings
class MySettings(Settings):
web = {
"with_ajax": False
}
api = Api('https://news.ycombinator.com/', settings=MySettings)
class Post(Item):
url = XPath('//a[@class="storylink"]/@href')
title = XPath('//a[@class="storylink"]/text()')
class Meta:
source = XPath('//tr[@class="athing"]')
route = {'/news?page=:page':'/news?p=:page'}
class Page(Item):
next_page = XPath('//a[@class="morelink"]/@href')
class Meta:
source = None
route = {'/news?page=:page':'/news?p=:page'}
def clean_next_page(self, next_page):
return "http://127.0.0.1:5000/" + str(next_page)
api.register(Post)
api.register(Page)
api.serve()
# Visit: http://127.0.0.1:5000/
- Send a single request to source web site with the same url.
- Fetch most of the data fetched from cache and storage.
- Get HTML from storage when the cache expired.
- Get HTML from source site when the storage expired.
$ pip install toapi
$ toapi -v
toapi, version 0.1.12
$ toapi new api
2017/12/14 09:16:54 [New project] OK Creating project directory "api"
Cloning into 'api'...
remote: Counting objects: 10, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 10 (delta 1), reused 10 (delta 1), pack-reused 0
Unpacking objects: 100% (10/10), done.
Checking connectivity... done.
2017/12/14 09:16:56 [New project] OK Success!
cd api
toapi run
Turn to the directory 'api' which you've just created. Run the following command:
$ toapi run
2017/12/14 09:27:18 [Serving ] OK http://127.0.0.1:5000
Now everything is done. Open http://127.0.0.1:5000 in your browser to have a look!
A Toapi app is a flask app. For deployment of Toapi, refer to Flask documentaion:
While lightweight and easy to use, Flask’s built-in server is not suitable for production as it doesn’t scale well and by default serves only one request at a time. Some of the options available for properly running Flask in production are documented here.
If you want to deploy your Flask application to a WSGI server not listed here, look up the server documentation about how to use a WSGI app with it. Just remember that your Flask application object is the actual WSGI application.
Deployment Options — Flask Documentation (0.12)
$ toapi new toapi/toapi-pic
$ cd toapi-pic
$ toapi run
# http://127.0.0.1:5000/_items
{
"/pic/?q=:key": [
"Pixabay",
"Pexels"
]
}
# http://127.0.0.1:5000/_status
{
"cache_get": 2,
"cache_set": 2,
"received": 4,
"sent": 2,
"storage_get": 1,
"storage_set": 2
}
# http://127.0.0.1:5000/pic/?q=coffee
{
"Pixabay": [
{
"img": "https://cdn.pixabay.com/photo/2017/06/21/05/28/coffee-2426110__340.png"
},
{
"img": "/static/img/blank.gif"
}
],
"Pexels": [
{
"img": "https://images.pexels.com/photos/302899/pexels-photo-302899.jpeg?h=350&auto=compress&cs=tinysrgb"
},
{
"img": "https://images.pexels.com/photos/34085/pexels-photo.jpg?h=350&auto=compress&cs=tinysrgb"
}
]
}
Toapi use cache to prevent repeated parsing and use storage to prevent sending request.
A toapi app has an ability to gather pages of multiple websites and convert them to easy to use APIs
Any application created by toapi could be shared to others.
A toapi app is a standard flask app, so that you can deploy your app as deploying a flask app.
A toapi app will automatically count kinds of states of itself and you can visit the states whenever you want.
To get help with Toapi, please use the GitHub issues
- Checking system every time running the app.