I recently moved my production app from Dokku to Kamal and loving it so far but one feature of rails i’ve been using went broken - the ActionCable support.
Basically, after the migration, my frontend is not able to connect to /cable endpoint anymore. The message in browser log says: WebSocket connection to 'wss://domain.com/cable' failed: without giving me any more information about why it failed after the colon. Also, the rails log is empty for /cable route. The only suspicious thing i can see in traefik logs are these lines:
But i’m not sure if that’s related as this app is on production already with decent traffic so i’m not able to tell if that relates to the failing wss connection or not. However, the amount of these logs doesn’t match the amount of failed wss connection retries so i would say that’s not it.
My kamal config looks as follows:
service: myapp
image: mbajur/myapp
volumes:
- "/home/app/myapp-cache:/app/tmp/cache"
- "/home/app/myapp-shared:/app/shared"
- "/home/app/myapp-storage:/app/storage"
servers:
web:
hosts:
- x.x.x.x
labels:
traefik.http.routers.myapp.entrypoints: websecure
traefik.http.routers.myapp.rule: Host(`domain.com`)
traefik.http.routers.myapp.tls.certresolver: letsencrypt
options:
network: "private"
job:
hosts:
- x.x.x.x
cmd: bundle exec rake solid_queue:start
options:
network: "private"
clock:
hosts:
- x.x.x.x
cmd: bundle exec clockwork clock.rb
options:
network: "private"
registry:
server: ghcr.io
username: mbajur
password:
- KAMAL_REGISTRY_PASSWORD
# Inject ENV variables into containers (secrets come from .env).
# Remember to run `kamal env push` after making changes!
env:
clear:
HOSTNAME: domain.com
APP_DOMAIN: domain.com
DB_HOST: x.x.x.x
RAILS_SERVE_STATIC_FILES: true
RAILS_LOG_TO_STDOUT: true
ARTISTS_TAXONOMY_ID: 9
CATEGORIES_TAXONOMY_ID: 8
PATTERNS_TAXONOMY_ID: 10
FLIPPER_PSTORE_PATH: shared/flipper.pstore
secret:
- POSTGRES_PASSWORD
- RAILS_MASTER_KEY
ssh:
user: app
builder:
dockerfile: Dockerfile.production
multiarch: false
cache:
type: registry
accessories:
db:
image: postgres:15
host: x.x.x.x
port: 5432
env:
clear:
POSTGRES_USER: "myapp"
POSTGRES_DB: 'myapp_production'
secret:
- POSTGRES_PASSWORD
files:
- config/init.sql:/docker-entrypoint-initdb.d/setup.sql
directories:
- data:/var/lib/postgresql/data
options:
network: "private"
traefik:
args:
accesslog: true
options:
network: "private"
publish:
- "443:443"
volume:
- "/letsencrypt/acme.json:/letsencrypt/acme.json"
args:
entryPoints.web.address: ":80"
entryPoints.websecure.address: ":443"
entryPoints.web.http.redirections.entryPoint.to: websecure # We want to force https
entryPoints.web.http.redirections.entryPoint.scheme: https
entryPoints.web.http.redirections.entrypoint.permanent: true
certificatesResolvers.letsencrypt.acme.email: "email@example.com"
certificatesResolvers.letsencrypt.acme.storage: "/letsencrypt/acme.json" # Must match the path in `volume`
certificatesResolvers.letsencrypt.acme.httpchallenge: true
certificatesResolvers.letsencrypt.acme.httpchallenge.entrypoint: web
healthcheck:
path: /health/ready
port: 4000
max_attempts: 15
# Bridge fingerprinted assets, like JS and CSS, between versions to avoid
# hitting 404 on in-flight requests. Combines all files from new and old
# version inside the asset_path.
# asset_path: /rails/public/assets
# Configure rolling deploys by setting a wait time between batches of restarts.
# boot:
# limit: 10 # Can also specify as a percentage of total hosts, such as "25%"
# wait: 2
# Configure the role used to determine the primary_host. This host takes
# deploy locks, runs health checks during the deploy, and follow logs, etc.
#
# Caution: there's no support for role renaming yet, so be careful to cleanup
# the previous role on the deployed hosts.
# primary_role: web
# Controls if we abort when see a role with no hosts. Disabling this may be
# useful for more complex deploy configurations.
#
# allow_empty_roles: false
I’d be looking to see if any load balancers or firewalls are not properly configured to support web sockets. Or maybe your server instance is underpowered and responding too slowly.
The weirdest thing about that is that not any error is even raised anywhere. No exact errors in JS console, no errors in network tab nor nginx server. Is there any way to at least get a error code using some other tool, like curl for example?
My server has plenty of free resources so i don’t think that’s it
I’m not an expert in these things but I think the reason you’re not seeing any errors in the browser is because it’s sending the request to the proxy (Traefik), which accepts the connection then acts as a client, forwarding the request to your server. So Traefik is reporting error because the breakdown is happening between it and the rails app, not between the browser and it.
And the server isn’t throwing anything because it’s none the wiser that there’s even a problem. It’s either not ever getting the request (because of a firewall or something) or it’s simply replying but doing so too slowly.
I’d start looking at network configurations and seeing what could be blocking or slowing down the request.
Thank you for all the effort. I will try to speak with traefik people and see where it takes me. I will write here in cause i find any solution for that.
