💾 Archived View for gmi.runtimeterror.dev › tailscale-ssh-serve-funnel captured on 2024-08-18 at 17:40:13. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-08-18)
➡️ Next capture (2024-08-25)
-=-=-=-=-=-=-
2023-12-20
I've spent the past two years in love with Tailscale [1], which builds on the secure and high-performance Wireguard VPN protocol [2] and makes it really easy to configure and manage [3]. Being able to easily (and securely) access remote devices as if they were on the same LAN is pretty awesome to begin with, but Tailscale is packed with an ever-expanding set of features that can really help to streamline your operations too. Here are three of my favorites.
[2] secure and high-performance Wireguard VPN protocol
[3] really easy to configure and manage
Tailscale already takes care of issuing, rotating, and otherwise managing the Wireguard keys used for securing communications between the systems in your tailnet. Tailscale SSH [4] lets it do the same for your SSH keys as well. No more manually dropping public keys on systems you're setting up for remote access. No more scrambling to figure out how to get your private key onto your mobile device so you can SSH to a server. No more worrying about who has access to what. Tailscale can solve all those concerns for you - and it does it without impacting traditional SSH operations:
*With Tailscale, you can already connect machines in your network, and encrypt communications end-to-end from one point to another—and this includes, for example, SSHing from your work laptop to your work desktop. Tailscale also knows your identity, since that’s how you connected to your tailnet. When you enable Tailscale SSH, Tailscale claims port 22 for the Tailscale IP address (that is, only for traffic coming from your tailnet) on the devices for which you have enabled Tailscale SSH. This routes SSH traffic for the device from the Tailscale network to an SSH server run by Tailscale, instead of your standard SSH server. With Tailscale SSH, based on the ACLs in your tailnet, you can allow devices to connect over SSH and rely on Tailscale for authentication instead of public key authentication.*
All you need to advertise Tailscale SSH on a system is to pass the appropriate flag in the `tailscale up` command:
sudo tailscale up --ssh
To actually use the feature, though, you'll need to make sure that your Tailscale ACL permits access. The default "allow all" ACL does this:
{
"acls": [
// Allow all connections.
{ "action": "accept", "src": ["*"], "dst": ["*:*"] },
],
"ssh": [
// Allow all users to SSH into their own devices in check mode.
{
"action": "check",
"src": ["autogroup:member"],
"dst": ["autogroup:self"],
"users": ["autogroup:nonroot", "root"]
}
]
}
The `acls` block allows all nodes to talk to each other over all ports, and the `ssh` block will allow all users in the tailnet to SSH into systems they own (both as nonroot users as well as the `root` account) but only if they have reauthenticated to Tailscale within the last 12 hours (due to that `check` action).
Most of my tailnet nodes are tagged with a location (`internal`/`external`) instead of belonging to a single user, and the `check` action doesn't work when the source is a tagged system. So my SSH ACL looks a bit more like this:
{
"acls": [
{
// internal systems can SSH to internal and external systems
"action": "accept",
"users": ["tag:internal"],
"ports": [
"tag:internal:22",
"tag:external:22"
],
},
],
"tagOwners": {
"tag:external": ["group:admins"],
"tag:internal": ["group:admins"],
},
"ssh": [
{
// users can SSH to their own systems and those tagged as internal and external
"action": "check",
"src": ["autogroup:members"],
"dst": ["autogroup:self", "tag:internal", "tag:external"],
"users": ["autogroup:nonroot", "root"],
},
{
// internal systems can SSH to internal and external systems,
// but external systems can't SSH at all
"action": "accept",
"src": ["tag:internal"],
"dst": ["tag:internal", "tag:external"],
"users": ["autogroup:nonroot"],
},
],
}
This way, SSH connections originating from `internal` systems will be accepted, while those originating from untagged systems will have the extra check for tailnet authentication. You might also note that this policy prevents connections from tagged systems as the `root` user, requiring instead that the user log in with their own account and then escalate as needed.
These ACLs can get pretty granular [5], and I think it's pretty cool to be able to codify your SSH access rules in a centrally-managed policy instead of having to manually keep track of which keys are on which systems.
Once SSH is enabled on a tailnet node and the ACL rules are in place, you can SSH from a Tailscale-protected system to another as easily as `ssh [hostname]` and you'll be connected right away - no worrying about keys or fumbling to enter credentials. I think this is doubly cool when implemented on systems running in The Cloud; Tailscale provides the connectivity so I don't need to open up port 22 to the world.
ssh tsdemo Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.5.11-6-pve x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage Last login: Tue Dec 19 04:17:15 UTC 2023 from 100.73.92.61 on pts/3 john@tsdemo:~$
As a bonus, I can also open an SSH session from the Tailscale admin console [6]
That even works from mobile devices, too!
I've mentioned in the past [7] how impressed I was (and still am!) by the Caddy webserver [8] and how effortless it makes configuring a reverse proxy with automatic TLS. I've used it for a *lot* of my externally-facing projects.
Caddy is great, but it's not quite as easy to use for internal stuff - I'd need a public DNS record and inbound HTTP access in order for the ACME challenge to complete and a cert to be issued and installed, or I would have to manually create a certificate and load it in the Caddy config. That's probably not a great fit for wanting to proxy my Proxmox host [9]. And that is where the capabilities of Tailscale Serve [10] really come in handy.
*Tailscale Serve is a feature that allows you to route traffic from other devices on your Tailscale network (known as a tailnet) to a local service running on your device. You can think of this as sharing the service, such as a website, with the rest of your tailnet.*
<-- note -->
Tailscale Serve requires that the MagicDNS and HTTPS features be enabled on your Tailnet. You can learn how to turn those on here [11].
<-- /note -->
The general syntax is:
tailscale serve <target>
*`<target>` can be a file, directory, text, or most commonly the location to a service running on the local machine. The location to the location service can be expressed as a port number (e.g., `3000`), a partial URL (e.g., `localhost:3000), or a full URL including a path (e.g., `http://localhost:3000/foo`).*
The command also supports some useful flags:
--bg, --bg=false
Run the command as a background process (default false)
--http uint
Expose an HTTP server at the specified port
--https uint
Expose an HTTPS server at the specified port (default mode)
--set-path string
Appends the specified path to the base URL for accessing the underlying service
--tcp uint
Expose a TCP forwarder to forward raw TCP packets at the specified port
--tls-terminated-tcp uint
Expose a TCP forwarder to forward TLS-terminated TCP packets at the specified port
--yes, --yes=false
Update without interactive prompts (default false)
Tailscale Serve can be used for spawning a simple file server (like this one which shares the contents of the `/demo` directory):
sudo tailscale serve /demo Available within your tailnet: https://tsdemo.tailnet-name.ts.net/ |-- path /demo Press Ctrl+C to exit.
Note that this server is running in the foreground, and that it's serving the site with an automatically-generated automatically-trusted Let's Encrypt [12] certificate.
I can also use Tailscale Serve for proxying another web server, like Cockpit [13], which runs on `http://localhost:9090`:
sudo tailscale serve --bg 9090 Available within your tailnet: => https://cockpit-project.org/ [13] Cockpit https://tsdemo.tailnet-name.ts.net/ |-- proxy http://127.0.0.1:9090 Serve started and running in the background. To disable the proxy, run: tailscale serve --https=443 off
This time, I included the `--bg` flag so that the server would run in the background, and I told it to proxy port `9090` instead of a file path.
But what if I want to proxy *another* service (like netdata [14], which runs on `http://localhost:19999`) at the same time? I can either proxy it on another port, like `8443`:
sudo tailscale serve --bg --https 8443 19999 Available within your tailnet: => https://github.com/netdata/netdata [14] netdata https://tsdemo.tailnet-name.ts.net/ |-- proxy http://127.0.0.1:9090 https://tsdemo.tailnet-name.ts.net:8443/ |-- proxy http://127.0.0.1:19999 Serve started and running in the background. To disable the proxy, run: tailscale serve --https=8443 off
Or serve it at a different path:
sudo tailscale serve --bg --set-path /netdata 19999 Available within your tailnet: https://tsdemo.tailnet-name.ts.net/ |-- proxy http://127.0.0.1:9090 https://tsdemo.tailnet-name.ts.net/netdata |-- proxy http://127.0.0.1:19999 Serve started and running in the background. To disable the proxy, run: tailscale serve --https=443 off
<-- note -->
Not all web apps adapt well to being served at a different path than they expect. It works fine for netadata, but did not work with Cockpit (at least not without digging deeper into the configuration to change the base URL). But hey, that's why we've got options!
<-- /note -->
So Tailscale Serve works well for making web resources available inside of my tailnet... but what if I want to share them externally? Tailscale Funnel [15] has that use case covered, and it makes it easy to share things without having to manage DNS records, punch holes in firewalls, or configure certificates.
*Tailscale Funnel is a feature that allows you to route traffic from the wider internet to a local service running on a machine in your Tailscale network (known as a tailnet). You can think of this as publicly sharing a local service, like a web app, for anyone to access—even if they don’t have Tailscale themselves.*
In addition to requiring that HTTPS is enabled, Funnel also requires that nodes have the `funnel` attribute applied through the ACL policy.
There's a default policy snippet to apply this:
"nodeAttrs": [
{
"target": ["autogroup:member"],
"attr": ["funnel"],
},
],
But I use a tag to manage funnel privileges instead so my configuration will look something like this:
{
"acls": [
{
// internal systems can SSH to internal and external systems
"action": "accept",
"users": ["tag:internal"],
"ports": [
"tag:internal:22",
"tag:external:22"
],
},
],
"tagOwners": {
"tag:external": ["group:admins"],
"tag:funnel": ["group:admins"],
"tag:internal": ["group:admins"],
},
"ssh": [
{
// users can SSH to their own systems and those tagged as internal and external
"action": "check",
"src": ["autogroup:members"],
"dst": ["autogroup:self", "tag:internal", "tag:external"],
"users": ["autogroup:nonroot", "root"],
},
{
// internal systems can SSH to internal and external systems,
// but external systems can't SSH at all
"action": "accept",
"src": ["tag:internal"],
"dst": ["tag:internal", "tag:external"],
"users": ["autogroup:nonroot"],
},
],
"nodeAttrs": [
{
// devices with the funnel tag can enable Tailscale Funnel
"target": ["tag:funnel"],
"attr": ["funnel"],
},
]
}
Now only nodes with the `funnel` tag will be able to enable Funnel.
From there, the process to activate Tailscale Funnel is basically identical to that of Tailscale Serve - you just use `tailscale funnel` instead of `tailscale serve`.
<-- note -->
A Funnel configuration is applied to the **port** that Tailscale Serve uses to make a resource available, not the resource itself. In the example above, I have both Cockpit and netdata being served over port `443`. If I try to use `sudo tailscale funnel --set-path /netdata 19999` to Funnel just the netdata instance, that will actually Funnel *both* resources instead of just the one.
<-- /note -->
If I want to make the netdata instance available publicly while keeping Cockpit internal-only, I'll need to serve netdata on a different port. Funnel only supports [16] ports `443`, `8443`, and `10000`, so I'll use `8443`:
sudo tailscale funnel --bg --https 8443 --set-path /netdata 19999 Available on the internet: => https://tailscale.com/kb/1223/funnel#limitations [16] only supports https://tsdemo.tailnet-name.ts.net:8443/netdata |-- proxy http://127.0.0.1:19999 Funnel started and running in the background. To disable the proxy, run: tailscale funnel --https=8443 off
It will take 10 or so minutes for the public DNS record to get created, but after that anyone on the internet (not just within my tailnet!) would be able to access the resource I've shared.
I can use `tailscale serve status` to confirm that both Cockpit and netdata are served internally on port `443`, but only netdata is published externally on port `8443`:
sudo tailscale serve status # Funnel on: # - https://tsdemo.tailnet-name.ts.net:8443 https://tsdemo.tailnet-name.ts.net (tailnet only) |-- / proxy http://127.0.0.1:9090 |-- /netdata proxy http://127.0.0.1:19999 https://tsdemo.tailnet-name.ts.net:8443 (Funnel on) |-- /netdata proxy http://127.0.0.1:19999
Pretty cool, right?
This post has covered some of my favorite (and most-used) Tailscale features, but there are *plenty* of other cool tricks up Tailscale's sleeves. These are some others that are definitely worth checking out:
golink](https://tailscale.com/blog/golink) (also covered [here
And if you're particularly nerdy like me, these will probably also grab your interest:
Manage Tailscale resources using Terraform
Need more inspiration? Tailscale has a pretty thorough collection of example uses cases [17] to help get the sparks flying.
[17] collection of example uses cases
I'm sure I'll have more Tailscale-centric posts [18] to share in the future, too. Stay tuned!
[18] more Tailscale-centric posts
---
Taking Taildrive for a Testdrive
Automate Packer Builds with GithHub Actions
Building Proxmox Templates with Packer
---