Build Up a Simple Backend with Express.js, Scrapy, PostgreSQL, and Heroku — Introduction

This series are recording every steps that how I build up a backend service with two programming languages, two frameworks, several libraries and a free could service. I will list down the main languages, frameworks and libraries.

There are too many things to introduce, so I will just write down something important thing that we should know from wikipedia or their document.

Node.js, Express.js, KNEX.js

Node.js is an open-source, cross-platform JavaScript run-time environment that executes JavaScript code outside of a browser.

Node.js

Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications.

Express - Node.js web application framework

Knex.js is a “batteries included” SQL query builder for Postgres, MSSQL, MySQL, MariaDB, SQLite3, Oracle, and Amazon Redshift designed to be flexible, portable, and fun to use.

Knex.js - A SQL Query Builder for Javascript

Python, Scarpy, beautifulsoup

Python is an interpreted, high-level, general-purpose programming language.

Welcome to Python.org

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

Scrapy | A Fast and Powerful Scraping and Web Crawling Framework

Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.

Beautiful Soup Documentation - Beautiful Soup 4.4.0 documentation

PostgreSQL, psycopg2

PostgreSQL, also known as Postgres, is a free and open-source relational database management system emphasizing extensibility and technical standards compliance. It is designed to handle a range of workloads, from single machines to data warehouses or Web services with many concurrent users.

PostgreSQL: The world's most advanced open source database

Psycopg is the most popular PostgreSQL database adapter for the Python programming language. Its main features are the complete implementation of the Python DB API 2.0 specification and the thread safety (several threads can share the same connection).

PostgreSQL + Python | Psycopg

Heroku

Heroku is a cloud platform as a service supporting several programming languages. Heroku, one of the first cloud platforms, has been in development since June 2007, when it supported only the Ruby programming language, but now supports Java, Node.js, Scala, Clojure, Python, PHP, and Go.

Cloud Application Platform | Heroku

These are the steps how I implement this project step by step.

1. Create a PostgreSQL with some datasheet.
2. Crawl data from a website with Python and Scrapy.
3. Store the data into PostgreSQL from the previous step.
4. Generate RESTful API by Node.js and Express.js.
5. Extract data from PostgreSQL with the previous step.
6. Put them all together and Deploy on Heroku.

All right, that all the outlines. I will create PostgreSQL in local database in the next article.

• Build Up a Simple Backend with Express.js, Scrapy, PostgreSQL, and Heroku — Introduction (you’re here)
• Build Up a Simple Backend with Express.js, Scrapy, PostgreSQL, and Heroku — PostgreSQL
• Build Up a Simple Backend with Express.js, Scrapy, PostgreSQL, and Heroku — Scrapy
• Build Up a Simple Backend with Express.js, Scrapy, PostgreSQL, and Heroku — Express.js
• Build Up a Simple Backend with Express.js, Scrapy, PostgreSQL, and Heroku — Heroku

laurence-liu/DearLiquorAPI.Express

laurence-liu/liquor.Scrapy