The robots exclusion standard defines a simple syntax to control access: User-agent: * Allow: / Such syntax can express well-known locations: Allow: /.well-known/feed --- syndication url Allow: /.well-known/host-meta -- RFC6415 Allow: /.well-known/icon -- favorite icon url Data encoded in text/gemini. To advertise capabilities. data:text/plain;charset=utf-8,User-agent%3A%20%2A%0D%0AAllow%3A%20%2F%0D%0A Allow%3A%20%2F.well-known%2Ffeed%0D%0AAllow%3A%20%2F.well-known%2Fhost-meta %0D%0AAllow%3A%20%2F.well-known%2Ficon data:text/plain;charset=utf-8;base64,VXNlci1hZ2VudDogKg0KQWxsb3c6IC8NCkFsbG 93OiAvLndlbGwta25vd24vZmVlZA0KQWxsb3c6IC8ud2VsbC1rbm93bi9ob3N0LW1ldGENCkFsb G93OiAvLndlbGwta25vd24vaWNvbg== A MAY, RFC wise. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20200604/5cab c9a5/attachment.htm>
---