💾 Archived View for gemini.wolffy.eu › gemlog › gemini.wolffy.eu › gemlog › 2024-08-24-Setting-up-Fl… captured on 2024-08-31 at 11:50:11. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-08-24)
-=-=-=-=-=-=-
Fluentd is a data collector and unification tool which helps me map the data from Molly-Brown's Error and Access logs to Prometheus and Grafana.
Sadly Fluentd doesn't come with Prometheus Support out of the Box so we have to build a custom image with the required Plugins preinstalled.
FROM fluentd:latest
USER root
RUN gem install fluent-plugin-prometheus
RUN rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
USER fluent
ENTRYPOINT ["tini", "--", "/bin/entrypoint.sh"]
CMD ["fluentd"]
To match and sort every line in our logs we need some custom Regexp for Both Access and Error logs.
<worker 0>
<source>
@type tail
path /fluentd/etc/logs/access.log
pos_file /fluentd/etc/pos/access.log.pos
tag molly.access
<parse>
@type regexp
expression /(?<time>[[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9][A-Z][0-9][0-9]:[0-9][0-9]:[0-9][0-9][A-Z]]*)\s(?<host>[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}]*)\s(?<code>[[0-9][0-9]]*)\s(?<request>[\S]*)/
time_format %Y-%m-%dT%H:%M:%SZ
</parse>
</source>
</worker>
<worker 1>
<source>
@type tail
path /fluentd/etc/logs/errors.log
pos_file /fluentd/etc/pos/errors.log.pos
tag molly.error
<parse>
@type regexp
expression /(?<time>[[0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9]\s[0-9][0-9]:[0-9][0-9]:[0-9][0-9]]*)\s(?<error>[\S ]*)/
time_format %Y/%m/%d %H:%M:%S
</parse>
</source>
</worker>
To integrate fluent neatly into my existing Gemini capsule docker compose and into my Grafana Network i had to append the following excert to my existing docker-compose.
fluent:
image: fluent:latest
container_name: fluent
volumes:
- config:/fluentd/etc
ports:
- "24231:24231"
networks:
- grafana_internal
networks:
grafana_internal:
external: true
Since this is my first time using fluent i was glad to find a helpfull medium.com post which i could use to guide me a little through my setup process.
The Github Readme of the Fluent Prometheus Plugin was also quite helpfull with the sample configuration.
fluent-plugin-prometheus Github
For all the Fluent buildin stuff like the regexp parser or the tail source the official docs were the answer to all my questions.
At the end my fluent config file looked something like this below.
Sadly at the time i'm writing this this is still not 100% working. All i get is the Access log in grafana but something with my regexp or the way i'm handling the error log still gives me headaches.
<system>
workers 3
</system>
<source>
@type forward
port 24224
bind 127.0.0.1
</source>
<worker 0>
<source>
@type tail
path /fluentd/etc/logs/access.log
pos_file /fluentd/etc/pos/access.log.pos
tag molly.access
<parse>
@type regexp
expression /(?<time>[[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9][A-Z][0-9][0-9]:[0-9][0-9]:[0-9][0-9][A-Z]]*)\s(?<host>[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}]*)\s(?<code>[[0-9][0-9]]*)\s(?<request>[\S]*)/
time_format %Y-%m-%dT%H:%M:%SZ
</parse>
</source>
</worker>
<worker 1>
<source>
@type tail
path /fluentd/etc/logs/errors.log
pos_file /fluentd/etc/pos/errors.log.pos
tag molly.error
<parse>
@type regexp
expression /(?<time>[[0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9]\s[0-9][0-9]:[0-9][0-9]:[0-9][0-9]]*)\s(?<error>[\S ]*)/
time_format %Y/%m/%d %H:%M:%S
</parse>
</source>
</worker>
<filter molly.error>
@type prometheus
<labels>
worker ${worker_id}
host ${hostname}
</labels>
<metric>
name molly_gemini_errors_total
type counter
desc Total number of errors
<labels>
error $.error
</labels>
</metric>
</filter>
<filter molly.access>
@type prometheus
<labels>
worker ${worker_id}
host ${hostname}
</labels>
<metric>
name molly_gemini_requests_total
type counter
desc Total number of requests
<labels>
host $.host
request $.request
code $.code
</labels>
</metric>
</filter>
<match molly.**>
@type prometheus
<labels>
host ${hostname}
</labels>
</match>
<match molly.access>
@type stdout
@id stdout_output_molly_access
</match>
<match molly.error>
@type stdout
@id stdout_output_molly_error
</match>
<source>
@type prometheus
bind 0.0.0.0
port 24231
metrics_path /metrics
</source>
<source>
@type prometheus_output_monitor
interval 10
<labels>
hostname ${hostname}
</labels>
</source>
Setting up the Grafana Dashboard wasnt as hard as i first thought. The grafana Visualization Builder is really helping a lot with creating Visualizations that fit your needs.
As stated before. The in my eyes most important thing to log which would be errors from inside Molly-Brown are sadly not usable right now.
molly_gemini_requests_total{code="<Status code>", host="<XXX.XXX.XXX.XXX>", instance="fluent:24231", job="gemini-capsule", request="gemini://<URI>", worker="0", worker_id="0"}
code = Response statuscode
host = IP Address of the
instance = fluent instance (Prometheus target)
job = Prometheus job name
request = requested URI
worker & worker_id = the fluent worker who managed the logging
value = the count of requests at start of fluent
molly_gemini_errors_total{error="<Error message>", instance="fluent:<Port>", job="gemini-capsule", worker="0", worker_id="0"}
error = Error message string as sent by Molly-Brown
instance = fluent instance (Prometheus target)
job = Prometheus job name
worker & worker_id = the fluent worker who managed the logging
value = the count of requests at start of fluent