falco is VCL parser and linter optimized for Fastly.

Go Reference Build

falco

falco is VCL parser and linter optimized for Fastly.

falco-demo

Disclaimer

This is a VCL parser, but dedicated to Fastly's VCL (version 2.x), so we don't care about the latest Varnish (7.x or later) syntax. The Varnish may have additional syntax, builtin function, predefined variables, but this tool may not parse correctly.

Additionally, Fastly provides its special builtin function, predefined variables. It's not compatible with Varnish. But this tool is optimized for them, we could parse and lint their declarations.

Motivation

Fastly is a really fantastic CDN, but sometimes we have problems with deployment operations. On deploy custom VCL to the Fastly, VCLs are validated when activating a new service version. Typically our deployment flow using custom VCLs is following:

  1. Clone active service and create new version
  2. Delete existing custom VCLs
  3. Upload new VCL files to the Fastly
  4. Activate new device version <= Validate VCLs on the Fastly cloud

Above flows take a time, and then if we have some mistakes on VCL e.g. missing semicolon X(, the deployment will fail. Additionally, unnecessary service versions will be created by our trivial issue.

To solve them, we made a Fastly dedicated VCL parser and linter tool to notice syntax errors and unexpected mistakes before starting the deployment flow.

Installation

Download binary from releases page according to your platform and place it under the $PATH.

Usage

Command help displays following:

falco -h
=======================================
  falco: Fastly VCL parser / linter
=======================================
Usage:
    falco [main vcl file]

Flags:
    -I, --include_path : Add include path
    -t, --transformer  : Specify transformer
    -h, --help         : Show this help
    -V, --version      : Display build version
    -v,                : Verbose warning lint result
    -vv,               : Verbose all lint result

Example:
    falco -I . -vv /path/to/vcl/main.vcl

Note:

Your VCL will have dependent modules loaded via include [module]. falco accept include path from -I, --include_path flag and search and load destination module from include path.

User defined subroutine

On linting, falco could not recognize when the user-defined subroutine is called, so you should apply the subroutine scope by adding annotation or its subroutine name. falco understands call scope by following rules:

Subroutine name

If the subroutine name has a suffix of _[scope], falco lint within that scope.

sub custom_recv { // name has `_recv` suffix, lint with RECV scope
  ...
}

sub custom_fetch { // name has `_fetch` suffix, lint with FETCH scope
  ...
}

Following table describes subroutine name and recognizing scope:

suffix scope example
_recv RECV sub custom_recv {}
_miss MISS sub custom_miss {}
_hash HASH sub custom_hash {}
_pass PASS sub custom_pass {}
_fetch FETCH sub custom_fetch {}
_error ERROR sub custom_error {}
_deliver DELIVER sub custom_deliver {}
_log LOG sub custom_log {}

Annotation

For some reasons, the subroutine name could not be changed. Then, if you apply a hint of scope on annotation, falco also understands scope:

// @recv
sub custom_process { // subroutine has `recv` annotation, lint with RECV scope
  ...
}

// @fetch
sub custom_request { // subroutine has `fetch` annotation, lint with FETCH scope
  ...
}

Following table describes annotation name and recognizing scope:

annotation scope example
@recv RECV // @recv
sub custom {}
@miss MISS // @miss
sub custom {}
@hash HASH // @hash
sub custom {}
@pass PASS // @pass
sub custom {}
@fetch FETCH // @fetch
sub custom {}
@error ERROR // @error
sub custom {}
@deliver DELIVER // @deliver
sub custom {}
@log LOG // @log
sub custom {}

Fastly related features

Currently, we don't support snippets which are managed in Fastly:

  • Edge Dictionary
  • VCL Snippets
  • Log definitions
  • Etc

Above snippets will be injected to your VCL top or extracting FASTLY XXX macro, but this tool aims to run locally, not communicating with Fastly service. However, we're planning to solve them using the Fastly API.

Lint error

falco has built in lint rules. see rules in detail. falco may report lots of errors and warnings because falco lints with strict type checks, disallows implicit type conversions even VCL is fuzzy typed language.

Overriding Severity

To avoid them, you can override severity levels by putting a configuration file named .falcorc on working directory. the configuration file contents format is following:

## /path/to/working/directory/.falcorc
regex/matched-value-override: IGNORE
...

Format is simply a yaml key-value object. The key is rule name, see rules.md and value should be one of IGNORE, INFO, WARNING and ERROR, case insensitive.

In the above case, the rule of regex/matched-value-override reports INFO as default, but overrides IGNORE which does not report it.

Error Levels

falco reports three of severity on linting:

ERROR

VCL may cause errors on Fastly, or may cause unexpected behavior for actual works.

WARNING

VCL could work, but may have potential bug and cause unexpected behavior for actual works.

falco does not output warnings as default. To see them, run with -v option.

INFORMATION

VCL is fine, but we suggest you improve your VCL considering Fastly recommendation.

falco does not output information as default. To see them, run with -vv option.

Transforming

falco is planning to transpile Fastly VCL to the other programming language e.g Go (HTTP service), node.js (Lambda@Edge) to use temporal CDN instead of Fastly.

Contribution

  • Fork this repository
  • Customize / Fix problem
  • Send PR :-)
  • Or feel free to create issues for us. We'll look into it

License

MIT License

Contributors

Owner
Comments
  • Git resolver

    Git resolver

    Fixes #98

    This adds two new command line flags:

    • -git: Which controls if paths are treated as versioned paths
    • -repo: Which set the path to the git repository

    Now all include paths are parsed to extract the version (git version) if present. If present we resolve the file with git show rather than opening the file directly

    chose to use git cmd rather than an in memory git implementation because it is harder with authentication etc

  • Add support for PCRE

    Add support for PCRE

    Why? Falco (the go regexp pkg to be more accurate!) right now doesn't know how to parse PCRE regular expressions that Varnish uses. An example of that would be this file: https://github.com/varnishcache/varnish-devicedetect/blob/master/devicedetect.vcl

    Edit1: I created a small repo that uses ccgo compiler to translate the PCRE2 source code to Go. https://github.com/shadialtarsha/go-pcre

    Would love to hear your opinion about this

    Edit2: CI is green. I added a build step for pullrequest.yaml and it passed, then I removed it.

  • feat(lint): Adds support for subs with return type

    feat(lint): Adds support for subs with return type

    Extends falco to support subroutines with return values

    Fixes https://github.com/ysugimoto/falco/issues/8

    Signed-off-by: Sotiris Nanopoulos [email protected]

  • Fixes #42 - Linting `client.geo.city.utf8`

    Fixes #42 - Linting `client.geo.city.utf8`

    As discussed in #42, setting the split depth to 3 causes variables such as client.geo.city.utf8 to not be recognized.

    Signed-off-by: Sotiris Nanopoulos [email protected]

  • Allow req.backend to be read as a string

    Allow req.backend to be read as a string

    Fastly documents some curious behaviour for the req.backend variable:

    Upon reading in a STRING context, req.backend is converted to a STRING and prepended with the backend's share_key property, which is your service ID (for backends created via the API, CLI, or web interface), the string "fastlyshield" (for shield backends), or whatever you defined (for backends defined in VCL).

    So req.backend is typically a BACKEND but it can be cast to a STRING as well, unlike other BACKEND values! We make use of this behaviour in our code to track what backend has been used for debugging purposes, i.e.:

    set resp.http.Debug-Fastly-Backend = req.backend;
    

    Currently falco (understandably) prints a linter error when encountering this code

    🔥 [ERROR] resp.http.Debug-Fastly-Backend wants type STRING but assign BACKEND (operator/assignment)
    

    I'm honestly not sure what the best workaround for this would be. I think the big question is: do any other Fastly variables exhibit similar magical casting behaviour? If so, it would probably be worth expanding the predefined.yml specification to allow get and set fields to return an array or their different possible types.

    For now, however, I'm assuming (or hoping) that this isn't the case and we can instead construct a specific hack for req.backend. I've started this off by adding an additional check in lintAssignOperator for whether you're reading the req.backend identifier. For this to work consistently we'll need to copy the check to all other places where a value is read, such as in lintFunctionArguments.

    It's very messy work though and I'm wondering if anyone knows a better way to handle these values which can have multiple types. Is there anything in code already that supports this behaviour?

    P.S. Thanks for giving me write access to the repository; I've made this change a branch in the repository rather than our fork!

  • Support PCRE Regular Expressions

    Support PCRE Regular Expressions

    Why? Falco (the go regexp pkg to be more accurate!) right now doesn't know how to parse PCRE regular expressions that Varnish uses. An example of that would be this file: https://github.com/varnishcache/varnish-devicedetect/blob/master/devicedetect.vcl

  • Add support for subroutines operating in multiple scopes

    Add support for subroutines operating in multiple scopes

    The current way we set annotations only allows for subroutines to be used in a single vcl state function.

    Fastly is allows that and it also leads to copy-pasting functions with different annotations, which is not a good pattern.

    I only implemented them as annotations in the form @<state_name> instead of supporting them in the function name because I think it will be weird to have sub foo_recv_miss_deliver

    Fixes #84

    Signed-off-by: Sotiris Nanopoulos [email protected]

  • PR merge practice

    PR merge practice

    Hey @ysugimoto

    I wanted to see if we could decouple the release process from the pr merge process.

    The way we consume Falco is by having an fork of main that has some of our own CI/release timbits. In general we would like to try to have the main branch of our internal repo and this repo be the same but we have some PRs:

    • https://github.com/ysugimoto/falco/pull/75
    • https://github.com/ysugimoto/falco/pull/74

    That have been approved but not yet merged (presumably because you want to cut a release and that takes some time) but I think that we can decouple the two processes.

    This would allow us to consume main as it is. Also that would also help with merge conflicts etc.

  • [Feature] Add support for Fastly Rate Limit Variables

    [Feature] Add support for Fastly Rate Limit Variables

    Hey,

    We are interested in contributing support for https://developer.fastly.com/reference/vcl/declarations/ratecounter and all the other rate-limit variables that fastly offers.

  • [BUG] Cannot parse

    [BUG] Cannot parse "return (restart)" statements.

    Describe the problem The parser errors on trying to parse the return restart; / return (restart); statement

    VCL code that cause the problem return (restart);

    Screenshots

    💥  Undefined prefix expression parse of restart
    in /home/ryanolee/projects/XXX/vcl/XXX.vcl at line 69, position 17
     [...]
     69|        return (restart);
                             ^^^^^^^
    

    Expected behavior Fastly currently accepts the return (restart); syntax. Falco should probably accept this as equivalent of the restart; token if not

    Desktop (please complete the following information):

    • OS: Ubuntu 20.04
    • Shell: zsh

    Great job with is tool. It is really cool :raised_hands:

  • [Question] Recursive lint

    [Question] Recursive lint

    Is there a way to recursively lint all VCL files that belong to some repository, or directory? In my case, we don't care to lint the main.vcl file, as it does not exist.

    At my company, we package VCL files into a VCL Package that is managed by a CI/CD plugin that I wrote. This plugin manages these VCL Packages. It is similar to NPM or pip. Because of this, these VCL Packages do not contain a main.vcl file, rather they contain multiple VCL files that get managed by said plugin. My goal here is to use Falco to lint these VCL packages. I have tried passing multiple files to the I flag, but that does not seem to work.

  • [Feature] falco-ignore

    [Feature] falco-ignore

    Fastly VCL supports some hidden features such as map and callback that are undocumented. Falco does not support these functions which causes the lint to fail every time. I think it could be beneficial to include some way to tell Falco to ignore the next line, sub-routine, or file.

    For example, if you use the prettier linter for JS, you can tell prettier to ignore the next line like such:

    // prettier-ignore
    let positiveInts = test.map((value) => {
        if (value > 0) {
            return value
        }
    })
    

    We could potentially do the same with Falco:

    // falco-ignore
    map(var.test, test_callback, ",");
    

    This can also act as a bandaid to any unexpected bugs a developer might run into while using Falco. If a developer is experiencing some other linting issues that is causing their lint to fail, they can always tell Falco to skip the next like, sub-routine or file to resolve said issues until the linting bug can be patched.

  • [Feature] Git Based Resolver

    [Feature] Git Based Resolver

    The way we use VCL and manage dependencies at $company is by having them versioned in a separate git repository and I would like to build a git file resolver that allows you to resolve paths based on path:version and then use git show $REV:$FILE to get the file rather than just reading it from disk

    So the proposed changes are:

    • GitFileResolver that implements the Resolver interface
    • func NewGitFileResolvers(main string, c *Config) ([]Resolver, error) and switch between the two implementations based on the command line flag

    Addition of two command line flags:

    • -repo : Path to the git repo that contains the modules
    • -git : Use git to read versioned files that in the form of path/to/file/:version

    WDYT?

  • [BUG] cross-compile failing

    [BUG] cross-compile failing

    Describe the problem

    When I make a new release, accidentally CI process failed. I'm guessing our code includes platform dependency since https://github.com/ysugimoto/falco/pull/66 has been merged.

    On CI, use ubuntu-latest, linux-amd64:

    • linux: success
    • darwin-amd64: fail
    • darwin-arm64: fail

    It should be fixed to release successfully.

    VCL code that cause the problem N/A Screenshots Screen Shot 2022-07-21 at 18 11 28

    Expected behavior All platform binaries should be compiled and release successfully. A clear and concise description of what you expected to happen.

    Desktop (please complete the following information):

    • OS: Ubuntu (GitHub Actions)
    • Shell bash

    Additional context Add any other context about the problem here.

  • Implement vcl snippet

    Implement vcl snippet

    This PR supports parsing/linting VCL snippets managed in Fastly.

    I'm guessing VCL snippets behavior in Faslty activate phase:

    1. Extract macros of "#FASTLY xxx" with VCL snippets corresponds to phase (recv, init, etc...)
    2. resolve include "snippet::<snippet_name>" statement with registered snippets (regular or dynamic)
    3. Build integrated VCL file

    Therefore, I modified the CLI process:

    1. Factory all VCL snippets if remote flag is provided
    2. Parse main VCL file
    3. Visit AST recursively, find include statement and resolve it (include from a local file or fetched VCL snippets)
    4. Extract macro, find #FASTLY xxx comments and embed snippets that correspond to the exact phase
    5. Integrate to single VCL file (as main VCL), and lint it

    I marked TODO comment due to duplicated declaration in other packages, but I'll solve in another PR.

Related tags
Package implement reading and writing popular playlist formats: PLS, ASX, M3U, XSPF and others.

go-playlist ⚠️ WARNING The API is not stable yet and can change. Package playlist implement reading and writing popular playlist formats: PLS, ASX, M3

Oct 14, 2021
ffcommander - An easy frontend to FFmpeg and Imagemagick to automatically process video and manipulate subtitles.

% FFCOMMANDER(1) ffcommander 2.39 % Mikael Hartzell (C) 2018 % 2021 Name ffcommander - An easy frontend to FFmpeg and Imagemagick to automatically pro

May 9, 2022
Parse and demux MPEG Transport Streams (.ts) natively in GO

This is a Golang library to natively parse and demux MPEG Transport Streams (ts) in GO. WARNING: this library is not yet production ready. Use at your

Jan 9, 2023
Parse and generate m3u8 playlists for Apple HTTP Live Streaming (HLS) in Golang (ported from gem https://github.com/sethdeckard/m3u8)

go-m3u8 Golang package for m3u8 (ported m3u8 gem https://github.com/sethdeckard/m3u8) go-m3u8 provides easy generation and parsing of m3u8 playlists d

Nov 19, 2022
golang library to read and write various subtitle formats

libgosubs Golang library to read and write subtitles in the following formats Advanced SubStation Alpha v4 SRT TTML v1.0 - This is based on the spec p

Sep 27, 2022
Go bindings for libVLC and high-level media player interface
Go bindings for libVLC and high-level media player interface

Go bindings for libVLC 2.X/3.X/4.X and high-level media player interface. The package can be useful for adding multimedia capabilities to applications

Dec 31, 2022
Plays videos using Prometheus and Grafana, e.g. Bad Apple.
Plays videos using Prometheus and Grafana, e.g. Bad Apple.

prometheus_video_renderer Plays videos using Prometheus and Grafana, e.g. Bad Apple. Modes Currently 3 different modes are supported. Bitmap The bitma

Nov 30, 2022
RTSP 1.0 client and server library for the Go programming language

RTSP 1.0 client and server library for the Go programming language

Jan 3, 2023
Desktop application to download videos and playlists from youtube by simply copying its url.
Desktop application to download videos and playlists from youtube by simply copying its url.

tubemp3 Desktop application to download videos and playlists from youtube by simply copying its url. You just need to run tubemp3 and copy (CTRL + C)

Oct 25, 2022
A simple library to extract video and audio frames from media containers (based on libav).
A simple library to extract video and audio frames from media containers (based on libav).

Reisen A simple library to extract video and audio frames from media containers (based on libav, i.e. ffmpeg). Dependencies The library requires libav

Jan 2, 2023
Project to get Youtube video descriptions and search those videos as required

FamPayProject Project to get Youtube video descriptions and search those videos as required Prerequisities Postgres DB for persisting data Youtube Dat

Nov 5, 2021
golang library for mux and demux file containers such as mpeg-ts,mpeg-ps,flv

gomedia mpeg-ts,mpeg-ps,flv muxer/demuxer mpeg-ts muxer mpeg-ts demuxer mpeg-ps muxer mpeg-ps demuxer flv muxer flv demuxer mpeg-ps will be done in th

Jan 4, 2023
Synthetic media is a realistic transformation of audio and video using artificial intelligence.

Synthetic media is a realistic transformation of audio and video using artificial intelligence.

Nov 20, 2021
golang function that download a video from youtube, and convert it to a mp3 file using ffmpeg

echedwnmp3 echedwnmp3 is a function that download a video from youtube, and convert it to a mp3 file using ffmpeg example package main import(echedwn

Dec 7, 2021
lmmp3 is a little golang library that download a video from youtube, and convert it to a mp3 file using ffmpeg

lmmp3 lmmp3 is a function that download a video from youtube, and convert it to a mp3 file using ffmpeg You need to have installed ffmpeg in your syst

Aug 12, 2022
👾 Annie is a fast, simple and clean video downloader built with Go.
 👾 Annie is a fast, simple and clean video downloader built with Go.

?? Annie is a fast, simple and clean video downloader built with Go. Installation Prerequisites Install via go install Homebrew (macOS only) Arch Linu

Jun 1, 2022
[WIP] a very simple, tiny and intuitive ffmpeg wrapper with a cli interface for inspecting & transforming media files supported by the original ffmpeg software

About a very simple, tiny and intuitive ffmpeg wrapper with a cli interface for inspecting & transforming media files supported by the original ffmpeg

Oct 21, 2022
CLI program for SEO 301 & 302 url rewrites in fastly with more magento features soon to come

Magento-Fastly Table of Contents Magento-Fastly Development & Testing Install fastly cli Features Installation Usage Development & Testing To test thi

Oct 29, 2021
xlsx2mysql: An tool of helping your fastly generate SQL from Excel.

xlsx2mysql An tool of helping your fastly generate SQL from Excel 中文文档 Origin In order to convert Excel to MySQL and I made a tool to implement.But Wh

Nov 13, 2021
Demonstrating how you can take an action to your intrusions detected by Falco using OpenFaaS functions
Demonstrating how you can take an action to your intrusions detected by Falco using OpenFaaS functions

Kubernetes Response Engine powered by OpenFaaS Although Falco can be used to detect any intrusion attempts and sends alerts to channels according to t

Aug 22, 2022