💾 Archived View for silbernagel.dev › gemlog › elixir-fly-and-grafana-cloud captured on 2023-09-28 at 15:38:31. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-09-08)
-=-=-=-=-=-=-
__ __ __ __
___ _/ / ___ ____ ___ __ _____/ /__ ___ _/ /_/ /
/ _ `/ _ \/ _ `/ _ \/ _ \/ // / _ / -_) _ `/ __/ _ \
\_,_/_//_/\_,_/ .__/ .__/\_, /\_,_/\__/\_,_/\__/_//_/
/_/ /_/ /___/
Posted on Mon, May 01 2023
If you have an elixir app running on Fly.io, sending your telemetry data to Grafana cloud is easy.
Lets start with Logs.
In order to ship logs to Grafana Cloud, we need to setup another application called logshipper. It is all detailed in the documentation, but the TLDR; is:
mkdir logshipper
fly launch --no-deploy --image ghcr.io/superfly/fly-log-shipper:latest`
fly secrets set ORG=personal fly secrets set ACCESS_TOKEN=$(fly auth token)
fly secrets set LOKI_URL= fly secrets set LOKI_USERNAME= fly secrets set LOKI_PASSWORD=
[[services]] http_checks = [] internal_port = 8686
flyctl deploy
Once deployed, this should start sending all the logs from all your fly apps in the configured organization to Grafana Cloud.
More configuration options for the logshipper app can be found in the github repo
Also, I'd recommend shipping your logs in JSON format using something similar to the logger_json library.
Traces are fairly easy to send to Grafana Cloud Tempo, but we'll need to start by adding the OpenTelemetry libraries. Depending on your needs, you may leave off some of these.
# ./mix.exs
defp deps do
[
...
{:opentelemetry_exporter, "~> 1.0"},
{:opentelemetry, "~> 1.0"},
{:opentelemetry_api, "~> 1.0"},
{:opentelemetry_ecto, "~> 1.0"},
{:opentelemetry_liveview, "~> 1.0.0-rc.4"},
{:opentelemetry_phoenix, "~> 1.0"},
{:opentelemetry_cowboy, "~> 0.2"}
]
end
Now we need to add some configuration to the config/runtime.exs file.
# ./config/runtime.exs
if config_env() == :prod do
otel_auth = System.get_env("OTEL_AUTH") ||
raise """
OTEL_AUTH is a required variable
"""
config :opentelemetry_exporter,
otlp_protocol: :grpc,
otlp_traces_endpoint: System.fetch_env!("OTLP_ENDPOINT"),
otlp_headers: [{"Authorization", "Basic #{otel_auth}"}]
end
Next, setup the environment variables.
echo -n 'username:password' | base64`
(replace username and password with the actual vaules)
All of these values can be set with one command:
flyctl secrets set OTLP_ENDPOINT=https://your_endpoint OTEL_RESOURCE_ATTRIBUTES=your_datasource_name OTEL_AUTH=your_base64_encoded_string
After setting these values and deploying the application, traces should start showing in Grafana!
For Metrics, I really like to use Prometheus and I find the easiest way to get started with Prometheus is using prom_ex. PromEx provides excellent documentation and is worth reading, but a quick guide to get it working:
mix prom_ex.gen.config --datasource curl`
config :your_app, YourApp.PromEx,
metrics_server: [
port: System.get_env("PROM_PORT") || 9091,
path: "/metrics",
protocol: :http,
pool_size: 5
]
def start(_type, _args) do
children = [
YourAppWeb.Endpoint,
# PromEx should be started after the Endpoint, to avoid unnecessary error messages
YourApp.PromEx,
...
]
With all of that, localhost:9091/metrics should be available when the app is running
Next, the metrics just need to be exposed so that Fly can scrape them. As documented by Fly, just add the following to your fly.toml file:
[metrics] port = 9091 path = "/metrics"
Finally, setup the Prometheus data source in Grafana Cloud with the following properties:
You should now see all of the fly metrics and prom_ex defined metrics in Grafana!
I wrote this because I wanted to add observability to my Elixir/Phoenix apps that run on Fly and finding the information I needed was scattered throughout the docs. Hopefully others find this useful.
Happy Observing!