Robots, paywalls and API’s are at the forefront of web related news at the moment. In this article, we take an in-depth look at why these parts of the online world have become such a hot topic for debate.
What is an API?
An Application Programming Interface, (API), is a piece of software that enables two or more computers to interact with one another. API’s are incredibly useful for creating automations such as scheduled posts on social media or embedding content from another website onto your own.
APIs have been a part of the Web since the rise of Web 2.0 at the turn of the millennium. Historically their use was widespread, but as bad actors began to exploit API’s, restrictions on what they are able to do has limited their implementation.
The rise of bots, scrapers and AI.
APIs are the main gateway by which bots are able to access social media platforms and other websites. Since the introduction of APIs, web developers have been desperate to find ways to prevent spam bots whilst also allowing genuinely helpful applications to get through.
The explosion of AI development in recent years has reignited the debate surrounding APIs. Large Language Models (LLMs) such as OpenAI’s ChatGPT use API’s to scan large parts of a website to generate responses to questions. This collection of data from websites that forms an AI’s language model is known as a ‘corpus’.
Information is an extremely valuable resource in the modern age of the internet. Owners of large social media platforms such as Reddit and Twitter, (now known as ‘X’) have become increasingly concerned that the content and information created by their user base is being stolen by AI driven searches.
The main concern is that if users are able to get the same quality of content from AI searches, they will not visit the platform itself. The most obvious example of this currently is the Chrome browser extension ‘ChatGPT for YouTube’ that summarises YouTube videos for the user without the need for them to watch them.
As a result of this, there have been cases where companies have limited access to APIs, moved the service behind paywalls and made their platform inaccessible without an account. The most notable, recent case for this was Reddit’s decision to move API access behind a paywall, and the subsequent protests known as the ‘Reddit Blackouts’.
AI is shaping the future of the Internet, but not in the way that you think.
It is becoming increasingly clear that, as online platforms move to ensure that the valued content of their services is guarded against data scraping, the Internet as a whole will become increasingly ‘walled off’.
A ‘Walled Garden’ is a term given to Websites and other forms of technology to describe their accessibility. Whilst open garden websites and services allow users and API’s to access their content free of charge and without an account, walled gardens implement barriers to safeguard their content.
An Internet that is dominated by walled off websites will completely reshape how we access information as well as the variety of information readily available to us. A quick Google search now for example, demonstrates how reliant we have become on readily available websites.
The affix ‘Reddit’ has become an increasingly common addition to Google searches. There are a few reasons for this, one is to get around advertisements but the other, more interesting reason is that users are keen to find organic, human conversation and content that answers their question.
Google has introduced new features to their results pages to prepare for the fall in crawlable pages from platforms such as Twitter and Reddit. The most notable of these is the introduction of ‘perspectives’ that aims to replicate the community feel of Reddit and Twitter on Google’s own platforms such as Blogger and YouTube.
The death of ‘The Free Internet’.
The move to enclosed information spaces on the Internet will fundamentally change how we can use it. Supporters of the ‘Free Internet’, such as the technology and law community Masaar have raised concerns that this will be the final nail in the coffin for the project.
The Free Internet is an idea that all information should be free, readily available and publicly owned, that is to say it is not owned by any one company. One way which we can move towards this vision of the internet is through the use of decentralised technology.
Web3 signifies a rare opportunity to protect the original idea of the World Wide Web. The core of Web3 will always be focused on the rights of the individual through privacy, freedom of information and collective ownership.
