How to Summarize Substack Articles using ChatGPT Api and Node.js
Scraping and OpenAI's chat completions API - Edition #24
Hey, I'm Marco and welcome to my newsletter!
As a software engineer, I created this newsletter to share my first-hand knowledge of the development world. Each topic we will explore will provide valuable insights, with the goal of inspiring and helping all of you on your journey.
In this episode, I show you how you can summarize Substack articles using OpenAI and scraping, at the end of the article you will find a graphical interface through which you can summarize your favourite articles.
You can summarize the articles for free via this link: https://substack-summarizer.bubbleapps.io/version-test
👋 Introduction
Over time, I have subscribed to numerous newsletters, each of which offers interesting and valuable content. However, the volume has become excessive and it is impossible for me to read them all. To solve this problem, I decided to implement an article summary.
This tool allows you to paste the link to any article and provides a concise summary of the content. In addition, it can translate the summary into any language you prefer, making it easier and smoother to read.
I implemented a similar solution to summarize YouTube videos in the user's preferred language, you can retrieve the episode here, inside you will find a link to access the UI:
🛠️ Solution design
The idea is to use scraping and OpenAI's chat completions API as shown below:
The user initiates the process by entering the desired language and the URL of the Substack article they want to summarize and translate.
The client sends a request to the server using the GET /summarize-article API endpoint. This request includes the language and URL provided by the user.
The server performs web scraping on the provided Substack URL to retrieve the article content. This involves fetching the webpage and extracting the text of the article.
The Substack platform responds to the server's request by returning the full content of the specified article.
The server sends the content of the article together with the specified language to the OpenAI API. It asks OpenAI to generate a summary of the article and translate it into the desired language.
The OpenAI API processes the server's request, generating a summary of the article and translating it into the specified language. The results are then sent back to the server.
The server returns the result of the operation to the client.
The client receives the summarized content and displays it to the user.
👨💻 Let's implement together
You can download all the code shown directly from my Github repository: https://github.com/marcomoauro/substack-summarizer
We start as usual with the backend template for Node.js that I have shown you here:
Model
we start by creating the Article.js model, with the aim of retrieving the content by scraping and summarizing it via the OpenAI chat completions API:
Keep reading with a 7-day free trial
Subscribe to Implementing to keep reading this post and get 7 days of free access to the full post archives.