Introduction

For a while robots.txt has been used by cyber security professionals to help identify endpoints on a website. These files use to contain useful endpoints for hackers to explore but in todays world they are usually useless.

Apple App Site Association

Apple-app-site-association(AASA) is a file that exists on domains which associates the domain with an IOS application. On IOS 9 and later if you click a link the device will determine if it should be handled by safari or the IOS app. To do this we can use a universal link which will relate a web URL to the IOS app. Before you had to use URL schemes which looked something like (com.myApp://login). The problem with this is that if the IOS app isnt installed and you open the link in safari it will render to nothing.

Using a universal link we can simply use the link (example.com/login) and the device will determine if it should be opened in a browser or on the app. Universal links allow us to have a fallback webpage if a user does not have the app installed.

The New robots.txt

To make use of universal links owners must host an apple app site association file on their domain. This file is hosted at “ /.well-known/apple-app-site-association” and holds a list of paths shown below:

Whats interesting about this file is that it exposes a boat load of endpoints. You can use this file to find hidden endpoints that a crawler or directory brute force would miss.

Robots.txt is a thing of the past you should be looking for “/.well-known/apple-app-site-association” if you want to find hidden paths in the application.